Started by user Ritesh Agarwal [EnvInject] - Loading node environment variables. Building remotely on magma-sc1403-32g-24c (magma_slave) in workspace /data/workspace/magma_temp_job3 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Running Prebuild steps [magma_temp_job3] $ /bin/sh -xe /tmp/jenkins6576394760968014429.sh ++ echo magma + desc=magma + echo Desc: 7.1.0-1817 - magma Desc: 7.1.0-1817 - magma Success build forhudson.tasks.Shell@317e8f66 [description-setter] Description set: 7.1.0-1817 - magma Success build forhudson.plugins.descriptionsetter.DescriptionSetterBuilder@57cd7ebc Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/magma_temp_job3 # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision b02f0a6891dcb53fcebd2cfe350412947f1e06c5 (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f b02f0a6891dcb53fcebd2cfe350412947f1e06c5 > /usr/bin/git rev-list 1d48d12bd1f657c3a7b5f59832f5f09e52604af4 # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #256 jenkins-magma_temp_job3-256 # timeout=10 No emails were triggered. [magma_temp_job3] $ /bin/bash /tmp/jenkins299557271058878722.sh [global] username:root password:couchbase port:8091 index_port:9102 n1ql_port:8093 data_path:/data [servers] 1:172.23.110.64 2:172.23.110.65 3:172.23.110.66 4:172.23.110.67 5:172.23.110.68 [membase] rest_username:Administrator rest_password:password [magma_temp_job3] $ /bin/sh -xe /tmp/jenkins2048925240512361778.sh + echo '' + echo '##############################################' ############################################## + echo 'Installing dependencies from requirements.txt' Installing dependencies from requirements.txt + echo '##############################################' ############################################## + /opt/jython/bin/pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): futures==3.3.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): requests==2.24.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): urllib3==1.25.10 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): ruamel.yaml==0.16.12 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): six==1.15.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 5)) Cleaning up... + echo '#################### Done ####################' #################### Done #################### + echo '' + echo version=7.1.0-1817 version=7.1.0-1817 + mkdir tr_for_install + cd tr_for_install + git clone https://github.com/couchbase/testrunner.git Cloning into 'testrunner'... + cd testrunner + git checkout master Already on 'master' + git pull origin master From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. + py_executable=python3 + python3 scripts/new_install.py -i /tmp/magma_temp_job3.ini -p timeout=1200,get-cbcollect-info=True,version=7.1.0-1817,product=cb,debug_logs=False,url=,install_debug_info=true 2021-12-01 11:07:46,700 - root - WARNING - URL: is not valid, will use version to locate build 2021-12-01 11:07:46,701 - root - INFO - SSH Connecting to 172.23.110.64 with username:root, attempt#1 of 5 2021-12-01 11:07:46,854 - root - INFO - SSH Connected to 172.23.110.64 as root 2021-12-01 11:07:47,201 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:47,609 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:47,611 - root - INFO - SSH Connecting to 172.23.110.65 with username:root, attempt#1 of 5 2021-12-01 11:07:47,787 - root - INFO - SSH Connected to 172.23.110.65 as root 2021-12-01 11:07:48,160 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:48,542 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:48,544 - root - INFO - SSH Connecting to 172.23.110.66 with username:root, attempt#1 of 5 2021-12-01 11:07:48,699 - root - INFO - SSH Connected to 172.23.110.66 as root 2021-12-01 11:07:49,070 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:49,455 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:49,457 - root - INFO - SSH Connecting to 172.23.110.67 with username:root, attempt#1 of 5 2021-12-01 11:07:49,615 - root - INFO - SSH Connected to 172.23.110.67 as root 2021-12-01 11:07:50,025 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:50,413 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:50,414 - root - INFO - SSH Connecting to 172.23.110.68 with username:root, attempt#1 of 5 2021-12-01 11:07:50,591 - root - INFO - SSH Connected to 172.23.110.68 as root 2021-12-01 11:07:50,986 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:51,366 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:51,369 - root - INFO - SSH Connecting to 172.23.110.64 with username:root, attempt#1 of 5 2021-12-01 11:07:51,517 - root - INFO - SSH Connected to 172.23.110.64 as root 2021-12-01 11:07:51,866 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:52,272 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:52,275 - root - INFO - SSH Connecting to 172.23.110.65 with username:root, attempt#1 of 5 2021-12-01 11:07:52,438 - root - INFO - SSH Connected to 172.23.110.65 as root 2021-12-01 11:07:52,785 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:53,165 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:53,167 - root - INFO - SSH Connecting to 172.23.110.66 with username:root, attempt#1 of 5 2021-12-01 11:07:53,311 - root - INFO - SSH Connected to 172.23.110.66 as root 2021-12-01 11:07:53,665 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:54,053 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:54,055 - root - INFO - SSH Connecting to 172.23.110.67 with username:root, attempt#1 of 5 2021-12-01 11:07:54,202 - root - INFO - SSH Connected to 172.23.110.67 as root 2021-12-01 11:07:54,575 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:54,955 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:54,956 - root - INFO - SSH Connecting to 172.23.110.68 with username:root, attempt#1 of 5 2021-12-01 11:07:55,106 - root - INFO - SSH Connected to 172.23.110.68 as root 2021-12-01 11:07:55,465 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:07:55,843 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-01 11:07:55,844 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,881 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,881 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,901 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,901 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,902 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,902 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,903 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,904 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,905 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,906 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,907 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,908 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,909 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,909 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,911 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,911 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,912 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:07:55,912 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:07:55,914 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1817/couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm is live 2021-12-01 11:08:02,053 - root - INFO - Done with uninstall on 172.23.110.67. 2021-12-01 11:08:02,420 - root - INFO - Done with uninstall on 172.23.110.66. 2021-12-01 11:09:29,457 - root - INFO - Done with uninstall on 172.23.110.64. 2021-12-01 11:09:38,934 - root - INFO - Done with uninstall on 172.23.110.65. 2021-12-01 11:09:39,975 - root - INFO - Done with uninstall on 172.23.110.68. 2021-12-01 11:09:42,559 - root - INFO - running command.raw on 172.23.110.64: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:42,588 - root - INFO - command executed successfully with root 2021-12-01 11:09:44,796 - root - INFO - running command.raw on 172.23.110.64: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:44,824 - root - INFO - command executed successfully with root 2021-12-01 11:09:45,651 - root - INFO - running command.raw on 172.23.110.65: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:45,676 - root - INFO - command executed successfully with root 2021-12-01 11:09:47,711 - root - INFO - running command.raw on 172.23.110.65: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:47,736 - root - INFO - command executed successfully with root 2021-12-01 11:09:48,498 - root - INFO - running command.raw on 172.23.110.66: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:48,524 - root - INFO - command executed successfully with root 2021-12-01 11:09:49,196 - root - INFO - running command.raw on 172.23.110.66: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:49,222 - root - INFO - command executed successfully with root 2021-12-01 11:09:49,906 - root - INFO - running command.raw on 172.23.110.67: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:49,932 - root - INFO - command executed successfully with root 2021-12-01 11:09:50,744 - root - INFO - running command.raw on 172.23.110.67: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:50,769 - root - INFO - command executed successfully with root 2021-12-01 11:09:51,602 - root - INFO - running command.raw on 172.23.110.68: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:51,628 - root - INFO - command executed successfully with root 2021-12-01 11:09:52,319 - root - INFO - running command.raw on 172.23.110.68: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1817-centos7.x86_64.rpm 2021-12-01 11:09:52,344 - root - INFO - command executed successfully with root 2021-12-01 11:11:26,097 - root - INFO - Done with install on 172.23.110.64. 2021-12-01 11:11:26,097 - root - INFO - Waiting for couchbase to be reachable 2021-12-01 11:11:26,106 - root - INFO - running command.raw on 172.23.110.64: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.64 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-01 11:11:26,527 - root - INFO - Done with install on 172.23.110.65. 2021-12-01 11:11:26,527 - root - INFO - Waiting for couchbase to be reachable 2021-12-01 11:11:26,533 - root - INFO - command executed successfully with root 2021-12-01 11:11:26,536 - root - INFO - running command.raw on 172.23.110.65: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.65 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-01 11:11:26,538 - root - INFO - running command.raw on 172.23.110.64: rm -rf /data/* 2021-12-01 11:11:27,002 - root - INFO - command executed successfully with root 2021-12-01 11:11:27,009 - root - INFO - running command.raw on 172.23.110.65: rm -rf /data/* 2021-12-01 11:11:27,160 - root - INFO - Done with install on 172.23.110.66. 2021-12-01 11:11:27,160 - root - INFO - Waiting for couchbase to be reachable 2021-12-01 11:11:27,168 - root - INFO - running command.raw on 172.23.110.66: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.66 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-01 11:11:27,272 - root - INFO - Done with install on 172.23.110.67. 2021-12-01 11:11:27,273 - root - INFO - Waiting for couchbase to be reachable 2021-12-01 11:11:27,282 - root - INFO - running command.raw on 172.23.110.67: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.67 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-01 11:11:27,639 - root - INFO - command executed successfully with root 2021-12-01 11:11:27,644 - root - INFO - running command.raw on 172.23.110.66: rm -rf /data/* 2021-12-01 11:11:27,732 - root - INFO - command executed successfully with root 2021-12-01 11:11:27,736 - root - INFO - running command.raw on 172.23.110.67: rm -rf /data/* 2021-12-01 11:11:27,788 - root - INFO - command executed successfully with root 2021-12-01 11:11:27,788 - root - INFO - running command.raw on 172.23.110.66: chown -R couchbase:couchbase /data 2021-12-01 11:11:27,853 - root - INFO - command executed successfully with root 2021-12-01 11:11:27,853 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-01 11:11:28,306 - root - INFO - command executed successfully with root 2021-12-01 11:11:28,306 - root - INFO - running command.raw on 172.23.110.67: chown -R couchbase:couchbase /data 2021-12-01 11:11:28,412 - root - INFO - command executed successfully with root 2021-12-01 11:11:28,412 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-01 11:11:30,610 - root - INFO - Setting data_path: /data: status True 2021-12-01 11:11:31,421 - root - INFO - Done with install on 172.23.110.68. 2021-12-01 11:11:31,421 - root - INFO - Waiting for couchbase to be reachable 2021-12-01 11:11:31,431 - root - INFO - running command.raw on 172.23.110.68: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.68 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-01 11:11:31,614 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.66 2021-12-01 11:11:31,614 - root - INFO - pools/default params : memoryQuota=172436 2021-12-01 11:11:31,619 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-01 11:11:31,620 - root - INFO - /node/controller/setupServices params on 172.23.110.66: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-01 11:11:31,653 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-01 11:11:31,653 - root - INFO - settings/web params on 172.23.110.66:8091:port=8091&username=Administrator&password=password 2021-12-01 11:11:31,755 - root - INFO - --> status:True 2021-12-01 11:11:31,755 - root - INFO - Done with init on 172.23.110.66. 2021-12-01 11:11:31,781 - root - INFO - Done with cleanup on 172.23.110.66. 2021-12-01 11:11:31,902 - root - INFO - command executed successfully with root 2021-12-01 11:11:31,907 - root - INFO - running command.raw on 172.23.110.68: rm -rf /data/* 2021-12-01 11:11:32,151 - root - INFO - Setting data_path: /data: status True 2021-12-01 11:11:33,157 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.67 2021-12-01 11:11:33,157 - root - INFO - pools/default params : memoryQuota=172436 2021-12-01 11:11:33,162 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-01 11:11:33,162 - root - INFO - /node/controller/setupServices params on 172.23.110.67: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-01 11:11:33,195 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-01 11:11:33,196 - root - INFO - settings/web params on 172.23.110.67:8091:port=8091&username=Administrator&password=password 2021-12-01 11:11:33,290 - root - INFO - --> status:True 2021-12-01 11:11:33,290 - root - INFO - Done with init on 172.23.110.67. 2021-12-01 11:11:33,487 - root - INFO - Done with cleanup on 172.23.110.67. 2021-12-01 11:12:58,829 - root - INFO - command executed successfully with root 2021-12-01 11:12:58,830 - root - INFO - running command.raw on 172.23.110.65: chown -R couchbase:couchbase /data 2021-12-01 11:12:58,856 - root - INFO - command executed successfully with root 2021-12-01 11:12:58,856 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-01 11:13:01,648 - root - INFO - Setting data_path: /data: status True 2021-12-01 11:13:02,653 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.65 2021-12-01 11:13:02,653 - root - INFO - pools/default params : memoryQuota=172436 2021-12-01 11:13:02,658 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-01 11:13:02,658 - root - INFO - /node/controller/setupServices params on 172.23.110.65: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-01 11:13:02,691 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-01 11:13:02,691 - root - INFO - settings/web params on 172.23.110.65:8091:port=8091&username=Administrator&password=password 2021-12-01 11:13:02,773 - root - INFO - --> status:True 2021-12-01 11:13:02,773 - root - INFO - Done with init on 172.23.110.65. 2021-12-01 11:13:02,801 - root - INFO - Done with cleanup on 172.23.110.65. 2021-12-01 11:13:11,349 - root - INFO - command executed successfully with root 2021-12-01 11:13:11,349 - root - INFO - running command.raw on 172.23.110.68: chown -R couchbase:couchbase /data 2021-12-01 11:13:11,379 - root - INFO - command executed successfully with root 2021-12-01 11:13:11,379 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-01 11:13:14,344 - root - INFO - Setting data_path: /data: status True 2021-12-01 11:13:15,359 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.68 2021-12-01 11:13:15,359 - root - INFO - pools/default params : memoryQuota=172436 2021-12-01 11:13:15,365 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-01 11:13:15,365 - root - INFO - /node/controller/setupServices params on 172.23.110.68: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-01 11:13:15,400 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-01 11:13:15,400 - root - INFO - settings/web params on 172.23.110.68:8091:port=8091&username=Administrator&password=password 2021-12-01 11:13:15,499 - root - INFO - --> status:True 2021-12-01 11:13:15,499 - root - INFO - Done with init on 172.23.110.68. 2021-12-01 11:13:15,526 - root - INFO - Done with cleanup on 172.23.110.68. 2021-12-01 11:13:23,470 - root - INFO - command executed successfully with root 2021-12-01 11:13:23,471 - root - INFO - running command.raw on 172.23.110.64: chown -R couchbase:couchbase /data 2021-12-01 11:13:23,501 - root - INFO - command executed successfully with root 2021-12-01 11:13:23,501 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-01 11:13:26,384 - root - INFO - Setting data_path: /data: status True 2021-12-01 11:13:27,391 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.64 2021-12-01 11:13:27,391 - root - INFO - pools/default params : memoryQuota=172436 2021-12-01 11:13:27,396 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-01 11:13:27,396 - root - INFO - /node/controller/setupServices params on 172.23.110.64: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-01 11:13:27,430 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-01 11:13:27,430 - root - INFO - settings/web params on 172.23.110.64:8091:port=8091&username=Administrator&password=password 2021-12-01 11:13:27,528 - root - INFO - --> status:True 2021-12-01 11:13:27,528 - root - INFO - Done with init on 172.23.110.64. 2021-12-01 11:13:27,561 - root - INFO - Done with cleanup on 172.23.110.64. 2021-12-01 11:13:27,583 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-01 11:13:27,619 - root - INFO - cluster:C1 node:172.23.110.64:8091 version:7.1.0-1817-enterprise aFamily:inet services:['kv'] 2021-12-01 11:13:27,619 - root - INFO - cluster:C2 node:172.23.110.65:8091 version:7.1.0-1817-enterprise aFamily:inet services:['kv'] 2021-12-01 11:13:27,619 - root - INFO - cluster:C3 node:172.23.110.66:8091 version:7.1.0-1817-enterprise aFamily:inet services:['kv'] 2021-12-01 11:13:27,619 - root - INFO - cluster:C4 node:172.23.110.67:8091 version:7.1.0-1817-enterprise aFamily:inet services:['kv'] 2021-12-01 11:13:27,619 - root - INFO - cluster:C5 node:172.23.110.68:8091 version:7.1.0-1817-enterprise aFamily:inet services:['kv'] 2021-12-01 11:13:27,619 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-01 11:13:27,620 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-01 11:13:27,620 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-01 11:13:27,620 - root - INFO - INSTALL COMPLETED ON: 172.23.110.64 2021-12-01 11:13:27,620 - root - INFO - INSTALL COMPLETED ON: 172.23.110.65 2021-12-01 11:13:27,620 - root - INFO - INSTALL COMPLETED ON: 172.23.110.66 2021-12-01 11:13:27,620 - root - INFO - INSTALL COMPLETED ON: 172.23.110.67 2021-12-01 11:13:27,620 - root - INFO - INSTALL COMPLETED ON: 172.23.110.68 2021-12-01 11:13:27,620 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-01 11:13:27,620 - root - INFO - TOTAL INSTALL TIME = 341 seconds + status=0 + cd ../.. + rm -rf tr_for_install + git clone https://github.com/gradle/guides.git Cloning into 'guides'... [magma_temp_job3] $ /bin/sh -xe /tmp/jenkins8282771302773184452.sh + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest' Starting a Gradle Daemon, 33 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.2.1 Using Java_client :: 3.2.2 Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar scripts/ssh.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar scripts/eagles_all_around.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main scripts/install.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest > Task :compileJava warning: unknown enum constant javax.annotation.meta.When.MAYBE reason: class file for javax.annotation.meta.When not found Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. 1 warning > Task :testrunner Global Test input params: {'bucket_eviction_policy': 'fullEviction', 'bucket_storage': 'magma', 'cluster_name': 'magma_temp_job3', 'ini': '/tmp/magma_temp_job3.ini', 'num_nodes': 5, 'rerun': 'False'} Logs will be stored at /data/workspace/magma_temp_job3/logs/testrunner-21-Dec-01_11-13-54/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey' Test Input params: {'fts_nodes': '0', 'doc_size': '1536', 'ops_rate': '200000', 'key_type': 'RandomKey', 'bucket_type': 'membase', 'num_collections': '50', 'maxttl': '10', 'graceful': 'True', 'rerun': 'False', 'num_nodes': 5, 'doc_ops': 'create:update:delete:read', 'ramQuota': '102400', 'eviction_policy': 'fullEviction', 'sdk_timeout': '60', 'case_number': 1, 'num_indexes': '100', 'cluster_name': 'magma_temp_job3', 'batch_size': '1000', 'ini': '/tmp/magma_temp_job3.ini', 'num_buckets': '1', 'log_level': 'debug', 'bucket_storage': 'magma', 'rebl_ops_rate': '50000', 'bucket_eviction_policy': 'fullEviction', 'iterations': '5', 'logs_folder': '/data/workspace/magma_temp_job3/logs/testrunner-21-Dec-01_11-13-54/test_1', 'skip_cleanup': 'True', 'nodes_init': '3', 'pc': '10', 'num_items': '40000000', 'cbas_nodes': '0', 'assert_crashes_on_load': 'True', 'mutation_perc': '100', 'bucket_names': 'GleamBook', 'randomize_doc_size': 'False', 'index_nodes': '0', 'randomize_value': 'True', 'infra_log_level': 'debug', 'key_size': '18'} SystemTestMagma (aGoodDoctor.Hospital.Murphy) ... 2021-12-01 11:14:00,973 | infra | INFO | MainThread | [task_manager:__init__:13] Initiating TaskManager with 30 threads 2021-12-01 11:14:00,977 | infra | DEBUG | MainThread | [cluster:__init__:29] Initiating ServerTasks 2021-12-01 11:14:00,979 | test | INFO | MainThread | [basetestcase:log_setup_status:658] ========= Murphy setup started for test #1 SystemTestMagma ========= 2021-12-01 11:14:01,197 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:14:01,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:01,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:14:01,648 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:01,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:14:01,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:14:01,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:14:01,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:14:01,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:14:01,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:14:01,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:14:01,801 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 11:14:01,802 | test | INFO | MainThread | [basetestcase:log_setup_status:658] ========= BaseTestCase setup started for test #1 SystemTestMagma ========= 2021-12-01 11:14:01,805 | test | DEBUG | MainThread | [basetestcase:setUp:327] Cleaning up cluster 2021-12-01 11:14:02,200 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 120secs 2021-12-01 11:14:02,288 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-01 11:14:02,289 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 360secs 2021-12-01 11:14:02,305 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-01 11:14:02,308 | test | WARNING | MainThread | [basetestcase:tearDownEverything:583] CLEANUP WAS SKIPPED 2021-12-01 11:14:02,309 | infra | INFO | MainThread | [basetestcase:tearDownEverything:626] ========== tasks in thread pool ========== 2021-12-01 11:14:02,311 | infra | INFO | MainThread | [basetestcase:tearDownEverything:628] ========================================== 2021-12-01 11:14:02,312 | test | INFO | MainThread | [basetestcase:initialize_cluster:414] Initializing cluster : C1 2021-12-01 11:14:02,316 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:14:02,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:02,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:14:02,776 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:02,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:14:02,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:14:02,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:14:02,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:14:02,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:14:02,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:14:02,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:14:02,990 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:14:03,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:03,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:14:03,450 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:03,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:14:03,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:14:03,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:14:03,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:14:03,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:14:03,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:14:03,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:14:03,641 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.64 - Checking if couchbase-server is installed 2021-12-01 11:14:03,673 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.64 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:03,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:03,746 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.64 - Running systemd command 2021-12-01 11:14:03,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: systemctl stop couchbase-server.service 2021-12-01 11:14:05,739 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.64 port:8091 ssh_username:root 2021-12-01 11:14:05,740 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 11:14:05,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /data/* 2021-12-01 11:14:05,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:05,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-01 11:14:05,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:05,855 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:14:05,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:06,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:14:06,322 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:06,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:14:06,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:14:06,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:14:06,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:14:06,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:14:06,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:14:06,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:14:06,476 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.64 - Checking if couchbase-server is installed 2021-12-01 11:14:06,503 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.64 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:06,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:06,572 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.64 - Running systemd command 2021-12-01 11:14:06,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: systemctl start couchbase-server.service 2021-12-01 11:14:06,604 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.64 port:8091 ssh_username:root 2021-12-01 11:14:06,605 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0x9 type=client open_count=1 channel=[id: 0x72876e8d, 0.0.0.0/0.0.0.0:49257] timeout=300.0>) 2021-12-01 11:14:06,615 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:07,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:07,153 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0xa type=client open_count=1 channel=[id: 0xf1568f25, 0.0.0.0/0.0.0.0:37481] timeout=300.0>) 2021-12-01 11:14:09,161 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:09,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:09,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0xb type=client open_count=1 channel=[id: 0x0421a346, 0.0.0.0/0.0.0.0:54683] timeout=300.0>) 2021-12-01 11:14:11,197 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:11,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:11,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0xc type=client open_count=1 channel=[id: 0xcb5b38ab, 0.0.0.0/0.0.0.0:58711] timeout=300.0>) 2021-12-01 11:14:13,220 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:13,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:13,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-01 11:14:15,255 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 360secs 2021-12-01 11:14:15,267 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 11:14:15,273 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 11:14:15,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:15,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 11:14:15,752 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:15,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 11:14:15,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 11:14:15,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 11:14:15,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 11:14:15,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 11:14:15,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 11:14:15,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 11:14:15,911 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 11:14:16,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:16,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 11:14:16,355 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:16,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 11:14:16,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 11:14:16,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 11:14:16,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 11:14:16,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 11:14:16,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 11:14:16,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 11:14:16,456 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.65 - Checking if couchbase-server is installed 2021-12-01 11:14:16,471 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.65 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:16,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:16,517 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.65 - Running systemd command 2021-12-01 11:14:16,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: systemctl stop couchbase-server.service 2021-12-01 11:14:18,553 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.65 port:8091 ssh_username:root 2021-12-01 11:14:18,555 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 11:14:18,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /data/* 2021-12-01 11:14:18,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:18,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-01 11:14:18,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:18,651 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 11:14:18,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:19,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 11:14:19,121 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:19,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 11:14:19,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 11:14:19,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 11:14:19,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 11:14:19,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 11:14:19,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 11:14:19,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 11:14:19,276 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.65 - Checking if couchbase-server is installed 2021-12-01 11:14:19,303 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.65 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:19,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:19,375 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.65 - Running systemd command 2021-12-01 11:14:19,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: systemctl start couchbase-server.service 2021-12-01 11:14:19,411 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.65 port:8091 ssh_username:root 2021-12-01 11:14:19,411 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0xa1630016, 0.0.0.0/0.0.0.0:43236] timeout=300.0>) 2021-12-01 11:14:19,417 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:19,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:19,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0xda290cfa, 0.0.0.0/0.0.0.0:41773] timeout=300.0>) 2021-12-01 11:14:21,444 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:21,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:21,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0x3d54f4f5, 0.0.0.0/0.0.0.0:45356] timeout=300.0>) 2021-12-01 11:14:23,467 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:23,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:23,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0x19478796, 0.0.0.0/0.0.0.0:51331] timeout=300.0>) 2021-12-01 11:14:25,484 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:25,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:25,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0xc2fe3ab1, 0.0.0.0/0.0.0.0:34107] timeout=300.0>) 2021-12-01 11:14:27,500 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:27,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:27,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-01 11:14:29,530 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 360secs 2021-12-01 11:14:29,543 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 11:14:29,546 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 11:14:29,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:29,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 11:14:30,026 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:30,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 11:14:30,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 11:14:30,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 11:14:30,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 11:14:30,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 11:14:30,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 11:14:30,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 11:14:30,203 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 11:14:30,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:30,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 11:14:30,671 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:30,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 11:14:30,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 11:14:30,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 11:14:30,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 11:14:30,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 11:14:30,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 11:14:30,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 11:14:30,806 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.66 - Checking if couchbase-server is installed 2021-12-01 11:14:30,828 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.66 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:30,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:30,888 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.66 - Running systemd command 2021-12-01 11:14:30,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: systemctl stop couchbase-server.service 2021-12-01 11:14:33,275 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.66 port:8091 ssh_username:root 2021-12-01 11:14:33,276 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 11:14:33,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /data/* 2021-12-01 11:14:33,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:33,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-01 11:14:33,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:33,371 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 11:14:33,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:33,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 11:14:33,828 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:33,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 11:14:33,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 11:14:33,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 11:14:33,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 11:14:33,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 11:14:33,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 11:14:33,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 11:14:33,966 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.66 - Checking if couchbase-server is installed 2021-12-01 11:14:33,987 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.66 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:34,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:34,043 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.66 - Running systemd command 2021-12-01 11:14:34,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: systemctl start couchbase-server.service 2021-12-01 11:14:34,075 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.66 port:8091 ssh_username:root 2021-12-01 11:14:34,075 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0xbe5cf474, 0.0.0.0/0.0.0.0:35431] timeout=300.0>) 2021-12-01 11:14:34,081 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:34,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:34,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0x151a582c, 0.0.0.0/0.0.0.0:52202] timeout=300.0>) 2021-12-01 11:14:36,098 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:36,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:36,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0xbb3d648c, 0.0.0.0/0.0.0.0:59422] timeout=300.0>) 2021-12-01 11:14:38,111 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:38,122 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:38,122 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0xec8a722e, 0.0.0.0/0.0.0.0:34497] timeout=300.0>) 2021-12-01 11:14:40,128 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:40,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:40,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-01 11:14:42,266 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 360secs 2021-12-01 11:14:42,279 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 11:14:42,283 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-01 11:14:42,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:42,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-01 11:14:42,801 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:42,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-01 11:14:42,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-01 11:14:42,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-01 11:14:42,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-01 11:14:42,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-01 11:14:42,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-01 11:14:42,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-01 11:14:42,999 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-01 11:14:43,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:43,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-01 11:14:43,482 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:43,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-01 11:14:43,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-01 11:14:43,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-01 11:14:43,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-01 11:14:43,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-01 11:14:43,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-01 11:14:43,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-01 11:14:43,644 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.67 - Checking if couchbase-server is installed 2021-12-01 11:14:43,673 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.67 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:43,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:43,746 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.67 - Running systemd command 2021-12-01 11:14:43,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: systemctl stop couchbase-server.service 2021-12-01 11:14:45,862 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.67 port:8091 ssh_username:root 2021-12-01 11:14:45,864 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-01 11:14:45,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /data/* 2021-12-01 11:14:45,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:45,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-01 11:14:45,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:45,960 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-01 11:14:46,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:46,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-01 11:14:46,461 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:46,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-01 11:14:46,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-01 11:14:46,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-01 11:14:46,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-01 11:14:46,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-01 11:14:46,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-01 11:14:46,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-01 11:14:46,627 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.67 - Checking if couchbase-server is installed 2021-12-01 11:14:46,654 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.67 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:46,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:46,727 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.67 - Running systemd command 2021-12-01 11:14:46,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: systemctl start couchbase-server.service 2021-12-01 11:14:46,765 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.67 port:8091 ssh_username:root 2021-12-01 11:14:46,765 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 Got this failure java.net.ConnectException: Connection refused: /172.23.110.67:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x03727303, 0.0.0.0/0.0.0.0:41379] timeout=300.0>) 2021-12-01 11:14:46,769 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.67:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:46,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:46,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.67:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0x64fa0c93, 0.0.0.0/0.0.0.0:58259] timeout=300.0>) 2021-12-01 11:14:48,786 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.67:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:48,798 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:48,798 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.67:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x4dd60e09, 0.0.0.0/0.0.0.0:33921] timeout=300.0>) 2021-12-01 11:14:50,802 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.67:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:50,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:50,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.67:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x7610344e, 0.0.0.0/0.0.0.0:39587] timeout=300.0>) 2021-12-01 11:14:52,821 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.67:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:52,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:52,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-01 11:14:55,150 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 360secs 2021-12-01 11:14:55,163 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-01 11:14:55,165 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-01 11:14:55,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:55,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-01 11:14:55,654 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:55,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-01 11:14:55,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-01 11:14:55,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-01 11:14:55,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-01 11:14:55,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-01 11:14:55,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-01 11:14:55,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-01 11:14:55,857 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-01 11:14:55,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:56,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-01 11:14:56,323 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:56,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-01 11:14:56,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-01 11:14:56,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-01 11:14:56,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-01 11:14:56,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-01 11:14:56,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-01 11:14:56,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-01 11:14:56,460 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.68 - Checking if couchbase-server is installed 2021-12-01 11:14:56,480 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.68 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:56,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:56,536 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.68 - Running systemd command 2021-12-01 11:14:56,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: systemctl stop couchbase-server.service 2021-12-01 11:14:58,479 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.68 port:8091 ssh_username:root 2021-12-01 11:14:58,479 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-01 11:14:58,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /data/* 2021-12-01 11:14:58,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:58,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-01 11:14:58,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-01 11:14:58,571 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-01 11:14:58,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:14:58,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-01 11:14:59,072 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:14:59,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-01 11:14:59,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-01 11:14:59,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-01 11:14:59,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-01 11:14:59,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-01 11:14:59,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-01 11:14:59,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-01 11:14:59,224 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.68 - Checking if couchbase-server is installed 2021-12-01 11:14:59,250 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.68 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-01 11:14:59,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /opt/couchbase/VERSION.txt 2021-12-01 11:14:59,319 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.68 - Running systemd command 2021-12-01 11:14:59,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: systemctl start couchbase-server.service 2021-12-01 11:14:59,352 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.68 port:8091 ssh_username:root 2021-12-01 11:14:59,354 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0xf60925f2, 0.0.0.0/0.0.0.0:44716] timeout=300.0>) 2021-12-01 11:14:59,358 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:14:59,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:14:59,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x6bd72235, 0.0.0.0/0.0.0.0:44568] timeout=300.0>) 2021-12-01 11:15:01,375 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:15:01,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:15:01,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0x77b1cf4f, 0.0.0.0/0.0.0.0:40876] timeout=300.0>) 2021-12-01 11:15:03,391 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:15:03,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:15:03,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0x5f409ec4, 0.0.0.0/0.0.0.0:50277] timeout=300.0>) 2021-12-01 11:15:05,407 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-01 11:15:05,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L839 2021-12-01 11:15:05,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-01 11:15:07,749 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 360secs 2021-12-01 11:15:07,766 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-01 11:15:07,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> reset_cluster():L562 2021-12-01 11:15:07,778 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-01 11:15:17,782 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:15:17,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:18,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:15:18,263 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:18,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:15:18,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:15:18,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:15:18,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:15:18,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:15:18,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:15:18,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:15:18,448 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 11:15:18,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:18,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 11:15:18,924 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:18,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 11:15:18,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 11:15:18,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 11:15:18,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 11:15:19,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 11:15:19,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 11:15:19,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 11:15:19,062 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 11:15:19,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:19,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 11:15:19,545 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:19,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 11:15:19,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 11:15:19,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 11:15:19,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 11:15:19,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 11:15:19,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 11:15:19,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 11:15:19,661 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-01 11:15:19,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:20,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-01 11:15:20,155 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:20,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-01 11:15:20,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-01 11:15:20,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-01 11:15:20,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-01 11:15:20,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-01 11:15:20,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-01 11:15:20,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-01 11:15:20,262 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-01 11:15:20,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:20,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-01 11:15:20,760 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:20,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-01 11:15:20,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-01 11:15:20,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-01 11:15:20,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-01 11:15:20,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-01 11:15:20,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-01 11:15:20,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-01 11:15:20,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /data/* 2021-12-01 11:15:20,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: chown -R couchbase:couchbase /data 2021-12-01 11:15:20,990 | test | DEBUG | MainThread | [rest_client:set_data_path:2233] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-01 11:15:23,752 | test | DEBUG | MainThread | [rest_client:set_data_path:2236] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-01 11:15:23,755 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.64_8091 2021-12-01 11:15:23,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /data/* 2021-12-01 11:15:23,756 | infra | INFO | pool-3-thread-1 | [task:start_task:89] Thread 'node_init_task_172.23.110.64_8091' started 2021-12-01 11:15:23,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: chown -R couchbase:couchbase /data 2021-12-01 11:15:23,782 | infra | DEBUG | pool-3-thread-1 | [task:wait_until:133] Wait for expected condition to get satisfied 2021-12-01 11:15:23,789 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:23,789 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-01 11:15:23,819 | test | DEBUG | MainThread | [rest_client:set_data_path:2233] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-01 11:15:23,823 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:23,823 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-01 11:15:23,865 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:23,865 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-01 11:15:23,921 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:23,921 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-01 11:15:24,022 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:24,023 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-01 11:15:24,203 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:24,203 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-01 11:15:24,545 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:24,545 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-01 11:15:25,206 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:25,207 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-01 11:15:26,507 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:26,509 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-01 11:15:26,684 | test | DEBUG | MainThread | [rest_client:set_data_path:2236] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-01 11:15:26,684 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.65_8091 2021-12-01 11:15:26,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /data/* 2021-12-01 11:15:26,684 | infra | INFO | pool-3-thread-2 | [task:start_task:89] Thread 'node_init_task_172.23.110.65_8091' started 2021-12-01 11:15:26,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: chown -R couchbase:couchbase /data 2021-12-01 11:15:26,721 | infra | DEBUG | pool-3-thread-2 | [task:wait_until:133] Wait for expected condition to get satisfied 2021-12-01 11:15:26,730 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:26,730 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-01 11:15:26,752 | test | DEBUG | MainThread | [rest_client:set_data_path:2233] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-01 11:15:26,836 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:26,838 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-01 11:15:26,875 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:26,875 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-01 11:15:26,931 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:26,931 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-01 11:15:27,028 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:27,029 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-01 11:15:27,204 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:27,204 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-01 11:15:27,542 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:27,542 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-01 11:15:28,200 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:28,201 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-01 11:15:29,081 | test | DEBUG | pool-3-thread-1 | [task:call:5874] server: ip:172.23.110.64 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266823929856L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.64:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.64', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1817-enterprise', 'indexMemoryQuota': 512, 'uptime': u'80', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-01 11:15:29,082 | test | DEBUG | pool-3-thread-1 | [rest_client:set_service_mem_quota:936] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-01 11:15:29,095 | test | DEBUG | pool-3-thread-1 | [rest_client:set_indexer_storage_mode:960] settings/indexes params: storageMode=plasma 2021-12-01 11:15:29,111 | test | DEBUG | pool-3-thread-1 | [rest_client:init_cluster:847] settings/web params on 172.23.110.64:8091:password=password&port=8091&username=Administrator 2021-12-01 11:15:29,496 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:29,496 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-01 11:15:29,599 | test | DEBUG | MainThread | [rest_client:set_data_path:2236] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-01 11:15:29,601 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.66_8091 2021-12-01 11:15:29,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /data/* 2021-12-01 11:15:29,602 | infra | INFO | pool-3-thread-3 | [task:start_task:89] Thread 'node_init_task_172.23.110.66_8091' started 2021-12-01 11:15:29,625 | infra | DEBUG | pool-3-thread-3 | [task:wait_until:133] Wait for expected condition to get satisfied 2021-12-01 11:15:29,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: chown -R couchbase:couchbase /data 2021-12-01 11:15:29,628 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:29,630 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-01 11:15:29,660 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:29,660 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-01 11:15:29,664 | test | DEBUG | MainThread | [rest_client:set_data_path:2233] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-01 11:15:29,698 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:29,698 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-01 11:15:29,755 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:29,755 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-01 11:15:29,851 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:29,851 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-01 11:15:30,029 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:30,029 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-01 11:15:30,365 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:30,365 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-01 11:15:31,023 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:31,023 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-01 11:15:32,069 | test | DEBUG | pool-3-thread-2 | [task:call:5874] server: ip:172.23.110.65 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266667524096L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.65:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.65', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1817-enterprise', 'indexMemoryQuota': 512, 'uptime': u'70', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-01 11:15:32,071 | test | DEBUG | pool-3-thread-2 | [rest_client:set_service_mem_quota:936] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-01 11:15:32,084 | test | DEBUG | pool-3-thread-2 | [rest_client:set_indexer_storage_mode:960] settings/indexes params: storageMode=plasma 2021-12-01 11:15:32,099 | test | DEBUG | pool-3-thread-2 | [rest_client:init_cluster:847] settings/web params on 172.23.110.65:8091:password=password&port=8091&username=Administrator 2021-12-01 11:15:32,322 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,323 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-01 11:15:32,486 | test | DEBUG | MainThread | [rest_client:set_data_path:2236] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-01 11:15:32,487 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.67_8091 2021-12-01 11:15:32,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /data/* 2021-12-01 11:15:32,489 | infra | INFO | pool-3-thread-4 | [task:start_task:89] Thread 'node_init_task_172.23.110.67_8091' started 2021-12-01 11:15:32,512 | infra | DEBUG | pool-3-thread-4 | [task:wait_until:133] Wait for expected condition to get satisfied 2021-12-01 11:15:32,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: chown -R couchbase:couchbase /data 2021-12-01 11:15:32,515 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,515 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-01 11:15:32,539 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,539 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-01 11:15:32,549 | test | DEBUG | MainThread | [rest_client:set_data_path:2233] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-01 11:15:32,572 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,572 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-01 11:15:32,625 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,627 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-01 11:15:32,719 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,720 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-01 11:15:32,895 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:32,895 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-01 11:15:33,232 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:33,232 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-01 11:15:33,885 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:33,885 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-01 11:15:34,897 | test | DEBUG | pool-3-thread-3 | [task:call:5874] server: ip:172.23.110.66 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266885197824L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.66:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.66', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1817-enterprise', 'indexMemoryQuota': 512, 'uptime': u'58', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-01 11:15:34,898 | test | DEBUG | pool-3-thread-3 | [rest_client:set_service_mem_quota:936] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-01 11:15:34,911 | test | DEBUG | pool-3-thread-3 | [rest_client:set_indexer_storage_mode:960] settings/indexes params: storageMode=plasma 2021-12-01 11:15:34,928 | test | DEBUG | pool-3-thread-3 | [rest_client:init_cluster:847] settings/web params on 172.23.110.66:8091:password=password&port=8091&username=Administrator 2021-12-01 11:15:35,178 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,180 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-01 11:15:35,461 | test | DEBUG | MainThread | [rest_client:set_data_path:2236] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-01 11:15:35,463 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.68_8091 2021-12-01 11:15:35,464 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.64_8091 2021-12-01 11:15:35,464 | infra | INFO | pool-3-thread-5 | [task:start_task:89] Thread 'node_init_task_172.23.110.68_8091' started 2021-12-01 11:15:35,464 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.65_8091 2021-12-01 11:15:35,466 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.66_8091 2021-12-01 11:15:35,466 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.67_8091 2021-12-01 11:15:35,492 | infra | DEBUG | pool-3-thread-5 | [task:wait_until:133] Wait for expected condition to get satisfied 2021-12-01 11:15:35,496 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,497 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-01 11:15:35,525 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,526 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-01 11:15:35,561 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,562 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-01 11:15:35,615 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,615 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-01 11:15:35,707 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,709 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-01 11:15:35,882 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:35,884 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-01 11:15:36,220 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:36,220 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-01 11:15:36,877 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:36,877 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-01 11:15:37,756 | test | DEBUG | pool-3-thread-4 | [task:call:5874] server: ip:172.23.110.67 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266915176448L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.67:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.67', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1817-enterprise', 'indexMemoryQuota': 512, 'uptime': u'48', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-01 11:15:37,757 | test | DEBUG | pool-3-thread-4 | [rest_client:set_service_mem_quota:936] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-01 11:15:37,776 | test | DEBUG | pool-3-thread-4 | [rest_client:set_indexer_storage_mode:960] settings/indexes params: storageMode=plasma 2021-12-01 11:15:37,792 | test | DEBUG | pool-3-thread-4 | [rest_client:init_cluster:847] settings/web params on 172.23.110.67:8091:password=password&port=8091&username=Administrator 2021-12-01 11:15:37,936 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.68_8091 2021-12-01 11:15:38,171 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L138 2021-12-01 11:15:38,171 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-01 11:15:40,743 | test | DEBUG | pool-3-thread-5 | [task:call:5874] server: ip:172.23.110.68 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266776530944L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.68:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.68', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1817-enterprise', 'indexMemoryQuota': 512, 'uptime': u'38', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-01 11:15:40,744 | test | DEBUG | pool-3-thread-5 | [rest_client:set_service_mem_quota:936] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-01 11:15:40,755 | test | DEBUG | pool-3-thread-5 | [rest_client:set_indexer_storage_mode:960] settings/indexes params: storageMode=plasma 2021-12-01 11:15:40,770 | test | DEBUG | pool-3-thread-5 | [rest_client:init_cluster:847] settings/web params on 172.23.110.68:8091:password=password&port=8091&username=Administrator 2021-12-01 11:15:40,900 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:710] RAM quota was defined less than 100 MB: 2021-12-01 11:15:40,900 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:713] Node: 172.23.110.64: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 256114988 kB\n', 'MemAvailable: 259646652 kB\n', 'Buffers: 40 kB\n', 'Cached: 4454364 kB\n', 'SwapCached: 372 kB\n', 'Active: 1546812 kB\n', 'Inactive: 3351568 kB\n', 'Active(anon): 428804 kB\n', 'Inactive(anon): 356008 kB\n', 'Active(file): 1118008 kB\n', 'Inactive(file): 2995560 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4151664 kB\n', 'Dirty: 80 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 443340 kB\n', 'Mapped: 107076 kB\n', 'Shmem: 340788 kB\n', 'Slab: 720556 kB\n', 'SReclaimable: 302152 kB\n', 'SUnreclaim: 418404 kB\n', 'KernelStack: 27760 kB\n', 'PageTables: 11704 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 3160164 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 779660 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 2048 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 273408 kB\n', 'DirectMap2M: 10870784 kB\n', 'DirectMap1G: 258998272 kB\n'] 2021-12-01 11:15:40,901 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:713] Node: 172.23.110.65: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 255937496 kB\n', 'MemAvailable: 259597704 kB\n', 'Buffers: 40 kB\n', 'Cached: 4470784 kB\n', 'SwapCached: 384 kB\n', 'Active: 1529204 kB\n', 'Inactive: 3399516 kB\n', 'Active(anon): 427824 kB\n', 'Inactive(anon): 407444 kB\n', 'Active(file): 1101380 kB\n', 'Inactive(file): 2992072 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4173728 kB\n', 'Dirty: 56 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 458360 kB\n', 'Mapped: 105444 kB\n', 'Shmem: 377368 kB\n', 'Slab: 866172 kB\n', 'SReclaimable: 450812 kB\n', 'SUnreclaim: 415360 kB\n', 'KernelStack: 27504 kB\n', 'PageTables: 12336 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 3186320 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 779660 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 4096 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 259072 kB\n', 'DirectMap2M: 17176576 kB\n', 'DirectMap1G: 252706816 kB\n'] 2021-12-01 11:15:40,901 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:713] Node: 172.23.110.66: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 256095312 kB\n', 'MemAvailable: 259560528 kB\n', 'Buffers: 28 kB\n', 'Cached: 4534296 kB\n', 'SwapCached: 580 kB\n', 'Active: 1537848 kB\n', 'Inactive: 3436272 kB\n', 'Active(anon): 438188 kB\n', 'Inactive(anon): 440648 kB\n', 'Active(file): 1099660 kB\n', 'Inactive(file): 2995624 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4092676 kB\n', 'Dirty: 60 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 440312 kB\n', 'Mapped: 101688 kB\n', 'Shmem: 439016 kB\n', 'Slab: 655584 kB\n', 'SReclaimable: 254064 kB\n', 'SUnreclaim: 401520 kB\n', 'KernelStack: 27440 kB\n', 'PageTables: 11672 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 3317328 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 779660 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 0 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 271360 kB\n', 'DirectMap2M: 16115712 kB\n', 'DirectMap1G: 253755392 kB\n'] 2021-12-01 11:15:40,901 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:713] Node: 172.23.110.67: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 256329400 kB\n', 'MemAvailable: 259836584 kB\n', 'Buffers: 40 kB\n', 'Cached: 4325856 kB\n', 'SwapCached: 132 kB\n', 'Active: 1562548 kB\n', 'Inactive: 3222252 kB\n', 'Active(anon): 430044 kB\n', 'Inactive(anon): 210892 kB\n', 'Active(file): 1132504 kB\n', 'Inactive(file): 3011360 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4179932 kB\n', 'Dirty: 180 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 458972 kB\n', 'Mapped: 118788 kB\n', 'Shmem: 182004 kB\n', 'Slab: 610648 kB\n', 'SReclaimable: 247376 kB\n', 'SUnreclaim: 363272 kB\n', 'KernelStack: 27360 kB\n', 'PageTables: 11572 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 2988336 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 778636 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 8192 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 259072 kB\n', 'DirectMap2M: 21370880 kB\n', 'DirectMap1G: 248512512 kB\n'] 2021-12-01 11:15:40,901 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:713] Node: 172.23.110.68: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 256200432 kB\n', 'MemAvailable: 259774132 kB\n', 'Buffers: 40 kB\n', 'Cached: 4315464 kB\n', 'SwapCached: 1260 kB\n', 'Active: 1544584 kB\n', 'Inactive: 3211924 kB\n', 'Active(anon): 429680 kB\n', 'Inactive(anon): 215080 kB\n', 'Active(file): 1114904 kB\n', 'Inactive(file): 2996844 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4095276 kB\n', 'Dirty: 836 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 440656 kB\n', 'Mapped: 103072 kB\n', 'Shmem: 203688 kB\n', 'Slab: 769984 kB\n', 'SReclaimable: 346084 kB\n', 'SUnreclaim: 423900 kB\n', 'KernelStack: 27616 kB\n', 'PageTables: 11264 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 3075824 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 778636 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 4096 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 271360 kB\n', 'DirectMap2M: 18212864 kB\n', 'DirectMap1G: 251658240 kB\n'] 2021-12-01 11:15:40,901 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 11:15:40,901 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 11:15:40,903 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 11:15:40,903 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-01 11:15:40,904 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-01 11:15:40,907 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:15:41,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:41,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:15:41,388 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:41,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:15:41,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:15:41,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:15:41,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:15:41,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:15:41,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:15:41,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:15:41,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-01 11:15:41,562 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.64 port:8091 ssh_username:root: [] 2021-12-01 11:15:41,562 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 11:15:41,563 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-01 11:15:41,568 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 11:15:41,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:41,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 11:15:42,052 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:42,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 11:15:42,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 11:15:42,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 11:15:42,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 11:15:42,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 11:15:42,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 11:15:42,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 11:15:42,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-01 11:15:42,232 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.65 port:8091 ssh_username:root: [] 2021-12-01 11:15:42,233 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 11:15:42,233 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-01 11:15:42,236 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 11:15:42,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:42,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 11:15:42,697 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:42,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 11:15:42,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 11:15:42,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 11:15:42,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 11:15:42,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 11:15:42,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 11:15:42,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 11:15:42,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-01 11:15:42,815 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.66 port:8091 ssh_username:root: [] 2021-12-01 11:15:42,815 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 11:15:42,815 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-01 11:15:42,819 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-01 11:15:42,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:43,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-01 11:15:43,303 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:43,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-01 11:15:43,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-01 11:15:43,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-01 11:15:43,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-01 11:15:43,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-01 11:15:43,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-01 11:15:43,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-01 11:15:43,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-01 11:15:43,420 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.67 port:8091 ssh_username:root: [] 2021-12-01 11:15:43,420 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-01 11:15:43,421 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-01 11:15:43,424 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-01 11:15:43,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:15:43,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-01 11:15:43,930 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:15:43,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-01 11:15:43,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-01 11:15:43,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-01 11:15:43,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-01 11:15:44,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-01 11:15:44,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-01 11:15:44,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-01 11:15:44,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-01 11:15:44,082 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.68 port:8091 ssh_username:root: [] 2021-12-01 11:15:44,082 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-01 11:15:44,082 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-01 11:15:44,082 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:361] Wait for all the services to come up after change_env_vars update 2021-12-01 11:15:44,091 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> change_env_variables():L363 2021-12-01 11:15:44,091 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-01 11:15:54,092 | test | INFO | MainThread | [basetestcase:initialize_cluster:439] Cluster C1 initialized 2021-12-01 11:15:54,092 | test | INFO | MainThread | [basetestcase:log_setup_status:658] ========= BaseTestCase setup finished for test #1 SystemTestMagma ========= ############################################################ # # Step 1: Create a 3 node cluster # ############################################################ 2021-12-01 11:15:54,144 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.110.65,172.23.110.66]_OUT=[]_1638386154.12 2021-12-01 11:15:54,147 | infra | INFO | pool-3-thread-6 | [task:start_task:89] Thread 'Rebalance_task_IN=[172.23.110.65,172.23.110.66]_OUT=[]_1638386154.12' started 2021-12-01 11:15:54,147 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.110.65,172.23.110.66]_OUT=[]_1638386154.12 2021-12-01 11:15:54,148 | test | DEBUG | pool-3-thread-6 | [rest_client:add_node:1178] Adding remote node 172.23.110.65:8091 to cluster 172.23.110.64:8091 2021-12-01 11:16:05,678 | test | DEBUG | pool-3-thread-6 | [rest_client:add_node:1178] Adding remote node 172.23.110.66:8091 to cluster 172.23.110.64:8091 2021-12-01 11:16:09,954 | test | DEBUG | pool-3-thread-6 | [rest_client:rebalance:1431] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-01 11:16:10,013 | test | DEBUG | pool-3-thread-6 | [rest_client:rebalance:1436] Rebalance operation started 2021-12-01 11:16:10,015 | test | INFO | pool-3-thread-6 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1817-enterprise | 0.152801122394 | Cluster node | | 172.23.110.65 | None | | | <--- IN --- | | 172.23.110.66 | None | | | <--- IN --- | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-01 11:16:10,026 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-01 11:16:10,026 | test | INFO | pool-3-thread-6 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-01 11:16:10,042 | infra | DEBUG | pool-3-thread-6 | [task:check:411] Wait before next rebalance progress check 2021-12-01 11:16:10,045 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-01 11:16:10,045 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-01 11:16:20,059 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1649 2021-12-01 11:16:20,061 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-01 11:16:25,071 | test | INFO | pool-3-thread-6 | [task:check:384] Rebalance - status: none, progress: 100 2021-12-01 11:16:25,089 | test | INFO | pool-3-thread-6 | [task:check:443] Rebalance completed with progress: 100% in 15.0759999752 sec 2021-12-01 11:16:25,091 | infra | INFO | pool-3-thread-6 | [task:complete_task:104] Thread 'Rebalance_task_IN=[172.23.110.65,172.23.110.66]_OUT=[]_1638386154.12' completed 2021-12-01 11:16:25,151 | test | INFO | MainThread | [rest_client:set_metadata_purge_interval:2141] Successfully set Metadata Purge Interval to 0.04 ############################################################ # # Step 2: Create required buckets and collections. # ############################################################ 2021-12-01 11:16:25,153 | test | INFO | MainThread | [opd:create_required_buckets:49] Get the available memory quota 2021-12-01 11:16:25,220 | test | INFO | MainThread | [opd:create_required_buckets:57] Create CB buckets 2021-12-01 11:16:25,220 | test | DEBUG | MainThread | [bucket_ready_functions:async_create_bucket:1569] Creating bucket: GleamBookUsers0 2021-12-01 11:16:25,223 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: bucket_GleamBookUsers0_create_task 2021-12-01 11:16:25,223 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for bucket_GleamBookUsers0_create_task 2021-12-01 11:16:25,285 | infra | INFO | pool-3-thread-7 | [BucketOperations_Rest:create_bucket:344] Creating 'membase' bucket GleamBookUsers0 2021-12-01 11:16:25,285 | infra | DEBUG | pool-3-thread-7 | [BucketOperations_Rest:create_bucket:346] http://172.23.110.64:8091/pools/default/buckets with param: storageBackend=magma&fragmentationPercentage=50&replicaIndex=1&maxTTL=0&flushEnabled=1&durabilityMinLevel=none&compressionMode=passive&bucketType=membase&conflictResolutionType=seqno&name=GleamBookUsers0&replicaNumber=1&ramQuotaMB=102400&evictionPolicy=fullEviction 2021-12-01 11:16:25,321 | infra | DEBUG | pool-3-thread-7 | [BucketOperations_Rest:create_bucket:354] 0.04 seconds to create bucket GleamBookUsers0 2021-12-01 11:16:25,322 | infra | INFO | pool-3-thread-7 | [task:complete_task:104] Thread 'bucket_GleamBookUsers0_create_task' completed 2021-12-01 11:16:25,322 | test | DEBUG | MainThread | [bucket_ready_functions:create_bucket:1581] Wait for memcached to accept cbstats or any other bucket request connections 2021-12-01 11:16:25,339 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> create_bucket():L1583 2021-12-01 11:16:25,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-01 11:16:27,400 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-01 11:16:27,400 | test | INFO | MainThread | [opd:create_required_buckets:79] Rebalance Starts 2021-12-01 11:16:27,430 | test | DEBUG | MainThread | [rest_client:rebalance:1431] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-01 11:16:27,486 | test | DEBUG | MainThread | [rest_client:rebalance:1436] Rebalance operation started 2021-12-01 11:16:27,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-01 11:16:27,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1567 2021-12-01 11:16:27,509 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-01 11:16:30,530 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1649 2021-12-01 11:16:30,530 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-01 11:16:35,551 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1567 2021-12-01 11:16:35,551 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-01 11:16:38,561 | test | INFO | MainThread | [rest_client:monitorRebalance:1576] Rebalance done. Taken 11.0730001926 seconds to complete 2021-12-01 11:16:38,568 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1579 2021-12-01 11:16:38,569 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-12-01 11:16:43,569 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection0 2021-12-01 11:16:43,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:43,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:44,135 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection1 2021-12-01 11:16:44,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:44,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:44,696 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection2 2021-12-01 11:16:44,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:44,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:45,256 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection3 2021-12-01 11:16:45,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:45,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:45,821 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection4 2021-12-01 11:16:45,892 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:45,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:46,394 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection5 2021-12-01 11:16:46,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:46,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:46,961 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection6 2021-12-01 11:16:47,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:47,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:47,529 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection7 2021-12-01 11:16:47,589 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:47,589 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:48,091 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection8 2021-12-01 11:16:48,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:48,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:48,658 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection9 2021-12-01 11:16:48,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:48,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:49,230 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection10 2021-12-01 11:16:49,293 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:49,293 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:49,793 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection11 2021-12-01 11:16:49,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:49,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:50,357 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection12 2021-12-01 11:16:50,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:50,418 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:50,920 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection13 2021-12-01 11:16:50,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:50,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:51,480 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection14 2021-12-01 11:16:51,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:51,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:52,042 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection15 2021-12-01 11:16:52,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:52,109 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:52,614 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection16 2021-12-01 11:16:52,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:52,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:53,180 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection17 2021-12-01 11:16:53,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:53,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:53,746 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection18 2021-12-01 11:16:53,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:53,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:54,309 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection19 2021-12-01 11:16:54,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:54,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:54,869 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection20 2021-12-01 11:16:54,931 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:54,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:55,434 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection21 2021-12-01 11:16:55,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:55,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:55,996 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection22 2021-12-01 11:16:56,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:56,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:56,569 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection23 2021-12-01 11:16:56,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:56,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:57,132 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection24 2021-12-01 11:16:57,191 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:57,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:57,694 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection25 2021-12-01 11:16:57,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:57,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:58,260 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection26 2021-12-01 11:16:58,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:58,329 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:58,831 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection27 2021-12-01 11:16:58,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:58,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:59,401 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection28 2021-12-01 11:16:59,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:16:59,474 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:16:59,974 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection29 2021-12-01 11:17:00,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:00,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:00,539 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection30 2021-12-01 11:17:00,602 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:00,604 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:01,104 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection31 2021-12-01 11:17:01,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:01,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:01,670 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection32 2021-12-01 11:17:01,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:01,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:02,236 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection33 2021-12-01 11:17:02,298 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:02,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:02,799 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection34 2021-12-01 11:17:02,862 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:02,862 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:03,364 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection35 2021-12-01 11:17:03,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:03,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:03,923 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection36 2021-12-01 11:17:03,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:03,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:04,484 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection37 2021-12-01 11:17:04,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:04,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:05,048 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection38 2021-12-01 11:17:05,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:05,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:05,611 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection39 2021-12-01 11:17:05,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:05,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:06,174 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection40 2021-12-01 11:17:06,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:06,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:06,740 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection41 2021-12-01 11:17:06,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:06,806 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:07,308 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection42 2021-12-01 11:17:07,369 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:07,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:07,871 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection43 2021-12-01 11:17:07,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:07,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:08,461 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection44 2021-12-01 11:17:08,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:08,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:09,072 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection45 2021-12-01 11:17:09,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:09,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:09,640 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection46 2021-12-01 11:17:09,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:09,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:10,204 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection47 2021-12-01 11:17:10,265 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:10,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:10,766 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection48 2021-12-01 11:17:10,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:10,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:11,332 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection49 2021-12-01 11:17:11,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-01 11:17:11,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-01 11:17:11,898 | test | DEBUG | MainThread | [opd:create_required_collections:119] Collections list == ['VolumeCollection46', 'VolumeCollection47', 'VolumeCollection44', 'VolumeCollection45', 'VolumeCollection48', 'VolumeCollection49', 'VolumeCollection42', 'VolumeCollection43', 'VolumeCollection40', 'VolumeCollection41', '_default', 'VolumeCollection35', 'VolumeCollection36', 'VolumeCollection33', 'VolumeCollection34', 'VolumeCollection39', 'VolumeCollection37', 'VolumeCollection38', 'VolumeCollection7', 'VolumeCollection6', 'VolumeCollection5', 'VolumeCollection4', 'VolumeCollection3', 'VolumeCollection31', 'VolumeCollection2', 'VolumeCollection32', 'VolumeCollection1', 'VolumeCollection0', 'VolumeCollection30', 'VolumeCollection9', 'VolumeCollection8', 'VolumeCollection19', 'VolumeCollection24', 'VolumeCollection25', 'VolumeCollection22', 'VolumeCollection23', 'VolumeCollection28', 'VolumeCollection29', 'VolumeCollection26', 'VolumeCollection27', 'VolumeCollection20', 'VolumeCollection21', 'VolumeCollection13', 'VolumeCollection14', 'VolumeCollection11', 'VolumeCollection12', 'VolumeCollection17', 'VolumeCollection18', 'VolumeCollection15', 'VolumeCollection16', 'VolumeCollection10'] 2021-12-01 11:17:11,913 | test | DEBUG | MainThread | [rest_client:update_autofailover_settings:2037] settings/autoFailover params: failoverServerGroup=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&timeout=600&enabled=false&canAbortRebalance=false [ip:172.23.110.67 port:8091 ssh_username:root, ip:172.23.110.68 port:8091 ssh_username:root] 2021-12-01 11:17:11,957 | infra | INFO | MainThread | [BucketOperations_Rest:update_memcached_settings:373] Updating memcached properties 2021-12-01 11:17:11,957 | infra | DEBUG | MainThread | [BucketOperations_Rest:update_memcached_settings:374] http://172.23.110.64:8091/pools/default/settings/memcached/global with param: num_reader_threads=disk_io_optimized&num_writer_threads=disk_io_optimized&num_storage_threads=40 2021-12-01 11:17:11,983 | infra | DEBUG | MainThread | [BucketOperations_Rest:update_memcached_settings:380] Memcached settings updated ############################################################ # # Step 1: Create 40000000 items sequentially # ############################################################ 2021-12-01 11:17:11,986 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 11:17:12,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 11:17:12,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 11:17:12,476 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 11:17:12,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 11:17:12,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 11:17:12,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 11:17:12,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 11:17:12,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 11:17:12,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 11:17:12,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 11:17:12,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-12-01 11:17:12,714 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 ############################################################ # # Step 2: Create 40000000 items sequentially # ############################################################ ############################################################ # # RAM FootPrint: 948.73828125 # ############################################################ Read Start: 0 Read End: 0 Update Start: 0 Update End: 0 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 0 Create End: 40000000 Final Start: 0 Final End: 40000000 2021-12-01 11:17:13,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:13,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1638386234.22 2021-12-01 11:17:14,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:14,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1638386235.25 2021-12-01 11:17:15,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:15,272 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1638386236.27 2021-12-01 11:17:16,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:16,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1638386237.29 2021-12-01 11:17:17,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:17,364 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1638386238.36 2021-12-01 11:17:18,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:18,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1638386239.39 2021-12-01 11:17:19,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:19,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1638386240.45 2021-12-01 11:17:20,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:20,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1638386241.47 2021-12-01 11:17:21,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:21,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1638386242.49 2021-12-01 11:17:22,505 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:22,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1638386243.51 2021-12-01 11:17:23,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:23,520 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:24,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:24,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:25,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:25,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:26,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:26,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:27,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:27,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:28,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:28,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:29,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:29,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:30,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:30,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:31,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:31,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:32,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:32,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:33,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:33,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:34,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:34,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:35,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:35,762 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:36,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:36,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:37,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:37,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:38,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:38,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:39,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:39,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:40,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:40,918 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:42,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:42,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:43,257 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:43,257 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:44,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:44,270 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:45,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:45,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:46,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:46,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:47,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:47,311 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:48,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:48,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:49,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:49,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:50,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:50,352 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:51,367 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:51,367 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:52,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:52,380 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:53,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:53,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:54,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:54,408 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:55,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:55,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:56,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:56,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:57,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:57,448 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:58,469 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:58,469 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:17:59,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:17:59,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:18:00,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:18:00,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:18:01,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:18:01,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:18:02,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:18:02,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 11:18:03,535 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 11:18:03,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection44_8_1638386236.27 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection35_0_1638386244.52 Loader_GleamBookUsers0__default_VolumeCollection43_3_1638386241.47 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638386245.54 Loader_GleamBookUsers0__default_VolumeCollection45_7_1638386237.29 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638386246.55 Loader_GleamBookUsers0__default_VolumeCollection46_10_1638386234.22 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638386247.57 Loader_GleamBookUsers0__default_VolumeCollection47_9_1638386235.25 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638386248.58 Loader_GleamBookUsers0__default_VolumeCollection41_1_1638386243.51 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638386249.62 Loader_GleamBookUsers0__default_VolumeCollection48_6_1638386238.36 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638386250.67 Loader_GleamBookUsers0__default_VolumeCollection49_5_1638386239.39 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638386251.69 Loader_GleamBookUsers0__default_VolumeCollection40_2_1638386242.49 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638386252.7 Loader_GleamBookUsers0__default_VolumeCollection42_4_1638386240.45 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638386253.71 Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638386245.54 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638386254.73 Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638386246.55 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638386255.75 Loader_GleamBookUsers0__default_VolumeCollection35_0_1638386244.52 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638386256.76 Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638386247.57 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638386257.78 Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638386249.62 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638386258.8 Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638386251.69 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638386259.81 Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638386253.71 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638386260.83 Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638386250.67 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638386261.92 Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638386248.58 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638386263.24 Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638386252.7 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638386264.26 Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638386254.73 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638386265.27 Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638386255.75 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638386266.29 Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638386258.8 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638386267.3 Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638386257.78 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638386268.31 Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638386260.83 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638386269.33 Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638386256.76 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638386270.34 Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638386263.24 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638386271.35 Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638386259.81 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638386272.37 Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638386261.92 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638386273.38 Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638386264.26 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638386274.39 Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638386265.27 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638386275.41 Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638386267.3 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638386276.42 Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638386269.33 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638386277.44 Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638386271.35 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638386278.45 Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638386266.29 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638386279.47 Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638386273.38 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638386280.48 Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638386268.31 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638386281.5 Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638386270.34 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638386282.51 Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638386272.37 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638386283.52 Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638386275.41 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638386284.54 Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638386274.39 is completed! Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638386279.47 is completed! Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638386277.44 is completed! Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638386281.5 is completed! Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638386276.42 is completed! Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638386278.45 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638386283.52 is completed! Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638386280.48 is completed! Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638386282.51 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638386284.54 is completed! 2021-12-01 15:19:49,391 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-01 15:19:49,397 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 15:19:49,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:19:49,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 15:19:49,855 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:19:49,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 15:19:49,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 15:19:49,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 15:19:49,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 15:19:49,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 15:19:49,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 15:19:49,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 15:19:50,010 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 15:19:50,012 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.01 2021-12-01 15:19:50,013 | infra | INFO | pool-3-thread-8 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.01' started 2021-12-01 15:19:50,015 | infra | INFO | pool-3-thread-8 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.01' completed 2021-12-01 15:19:50,017 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 15:19:50,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:19:50,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 15:19:50,480 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:19:50,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 15:19:50,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 15:19:50,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 15:19:50,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 15:19:50,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 15:19:50,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 15:19:50,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 15:19:50,631 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 15:19:50,632 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.63 2021-12-01 15:19:50,634 | infra | INFO | pool-3-thread-9 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.63' started 2021-12-01 15:19:50,635 | infra | INFO | pool-3-thread-9 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.63' completed 2021-12-01 15:19:50,638 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 15:19:50,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:19:51,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 15:19:51,118 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:19:51,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 15:19:51,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 15:19:51,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 15:19:51,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 15:19:51,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 15:19:51,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 15:19:51,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 15:19:51,243 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 15:19:51,243 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400791.24 2021-12-01 15:19:51,244 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.01 2021-12-01 15:19:51,244 | infra | INFO | pool-3-thread-10 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400791.24' started 2021-12-01 15:19:51,246 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 15:19:51,246 | infra | INFO | pool-3-thread-10 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400791.24' completed 2021-12-01 15:19:51,246 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400790.63 2021-12-01 15:19:51,247 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 15:19:51,247 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638400791.24 2021-12-01 15:19:51,249 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 15:19:51,289 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3946] Stats: {u'opsPerSec': 20018.2, u'diskFetches': 0, u'dataUsed': 4506574479816L, u'itemCount': 1999850916, u'quotaPercentUsed': 79.13226621349652, u'storageTotals': {u'hdd': {u'usedByData': 4555016243935L, u'used': 4866615740985L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 31109194249671L}, u'ram': {u'usedByData': 254902871584L, u'used': 332909170688L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 4555016243935L, u'vbActiveNumNonResident': 1860183007, u'memUsed': 254902871584L} 2021-12-01 15:19:51,651 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 1999850916 | 300.00 GiB | 237.40 GiB | 4.14 TiB | 6.98918700806 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ 2021-12-01 15:19:51,653 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+---------------+---------------+---------------+ | 0 | 2000000000 | 0-0 | 0-40000000 | 0-0 | 0-0 | +---------------+---------------+---------------+---------------+---------------+---------------+ 2021-12-01 15:19:52,023 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 6.98918700806 2021-12-01 15:19:52,026 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 15:19:52,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:19:52,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 15:19:52,499 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:19:52,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 15:19:52,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 15:19:52,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 15:19:52,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 15:19:52,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 15:19:52,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 15:19:52,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 15:19:52,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 15:19:53,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 15:19:53,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 15:19:53,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 15:19:53,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 15:19:54,799 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 15:19:54,802 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 15:19:54,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:19:55,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 15:19:55,296 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:19:55,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 15:19:55,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 15:19:55,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 15:19:55,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 15:19:55,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 15:19:55,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 15:19:55,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 15:19:55,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 15:19:55,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 15:19:55,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 15:19:55,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 15:19:56,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 15:19:57,341 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 15:19:57,345 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 15:19:57,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:19:57,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 15:19:57,828 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:19:57,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 15:19:57,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 15:19:57,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 15:19:57,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 15:19:57,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 15:19:57,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 15:19:57,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 15:19:57,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 15:19:58,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 15:19:58,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 15:19:58,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 15:19:58,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 15:19:59,947 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 15:19:59,948 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 4430238MB 2021-12-01 15:19:59,948 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 6974MB 2021-12-01 15:19:59,948 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 206798MB 2021-12-01 15:19:59,950 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 4223053MB 2021-12-01 15:19:59,954 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 15:20:00,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:20:00,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 15:20:00,424 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:20:00,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 15:20:00,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 15:20:00,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 15:20:00,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 15:20:00,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 15:20:00,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 15:20:00,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 15:20:00,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 15:20:00,655 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-01 15:20:00,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 15:20:01,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 15:20:01,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 15:20:02,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 15:20:02,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 15:20:02,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 15:20:03,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 15:20:03,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 15:20:04,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 15:20:04,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 15:20:05,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 15:20:05,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 15:20:05,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 15:20:06,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 15:20:06,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 15:20:07,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 15:20:07,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 15:20:08,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 15:20:08,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 15:20:09,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 15:20:09,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 15:20:10,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 15:20:10,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 15:20:10,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 15:20:11,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 15:20:11,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 15:20:11,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 15:20:12,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 15:20:12,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 15:20:13,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 15:20:13,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 15:20:14,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 15:20:14,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 15:20:14,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 15:20:15,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 15:20:15,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 15:20:16,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 15:20:16,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 15:20:17,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 15:20:17,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 15:20:18,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 15:20:18,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 15:20:19,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 15:20:19,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 15:20:20,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 15:20:20,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 15:20:21,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 15:20:21,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 15:20:21,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 15:20:22,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 15:20:22,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 15:20:23,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 15:20:23,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 15:20:23,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 15:20:24,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 15:20:24,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 15:20:25,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 15:20:25,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 15:20:26,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 15:20:26,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 15:20:27,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 15:20:27,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 15:20:27,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 15:20:28,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 15:20:28,816 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 15:20:28,819 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 15:20:28,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:20:29,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 15:20:29,299 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:20:29,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 15:20:29,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 15:20:29,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 15:20:29,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 15:20:29,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 15:20:29,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 15:20:29,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 15:20:29,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 15:20:29,476 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-01 15:20:29,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 15:20:29,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 15:20:30,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 15:20:30,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 15:20:31,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 15:20:31,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 15:20:31,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 15:20:32,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 15:20:32,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 15:20:32,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 15:20:33,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 15:20:33,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 15:20:34,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 15:20:34,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 15:20:35,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 15:20:35,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 15:20:36,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 15:20:36,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 15:20:36,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 15:20:37,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 15:20:37,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 15:20:38,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 15:20:38,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 15:20:39,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 15:20:39,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 15:20:39,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 15:20:40,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 15:20:40,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 15:20:41,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 15:20:41,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 15:20:42,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 15:20:42,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 15:20:42,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 15:20:43,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 15:20:43,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 15:20:44,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 15:20:44,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 15:20:45,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 15:20:45,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 15:20:46,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 15:20:46,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 15:20:47,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 15:20:47,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 15:20:47,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 15:20:48,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 15:20:48,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 15:20:48,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 15:20:49,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 15:20:49,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 15:20:50,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 15:20:50,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 15:20:50,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 15:20:51,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 15:20:51,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 15:20:51,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 15:20:52,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 15:20:52,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 15:20:53,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 15:20:53,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 15:20:54,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 15:20:54,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 15:20:55,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 15:20:55,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 15:20:55,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 15:20:56,354 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 15:20:56,357 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 15:20:56,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:20:56,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 15:20:56,809 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:20:56,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 15:20:56,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 15:20:56,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 15:20:56,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 15:20:56,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 15:20:56,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 15:20:56,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 15:20:57,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 15:20:57,081 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-01 15:20:57,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 15:20:57,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 15:20:58,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 15:20:58,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 15:20:58,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 15:20:59,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 15:20:59,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 15:21:00,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 15:21:00,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 15:21:01,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 15:21:01,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 15:21:02,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 15:21:02,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 15:21:03,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 15:21:03,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 15:21:04,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 15:21:04,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 15:21:05,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 15:21:05,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 15:21:06,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 15:21:06,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 15:21:07,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 15:21:07,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 15:21:08,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 15:21:08,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 15:21:09,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 15:21:09,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 15:21:10,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 15:21:10,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 15:21:11,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 15:21:11,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 15:21:12,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 15:21:12,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 15:21:13,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 15:21:13,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 15:21:13,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 15:21:14,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 15:21:14,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 15:21:15,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 15:21:15,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 15:21:16,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 15:21:16,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 15:21:17,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 15:21:17,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 15:21:18,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 15:21:18,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 15:21:19,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 15:21:19,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 15:21:20,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 15:21:20,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 15:21:21,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 15:21:21,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 15:21:22,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 15:21:22,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 15:21:23,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 15:21:23,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 15:21:23,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 15:21:24,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 15:21:24,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 15:21:25,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 15:21:25,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 15:21:26,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 15:21:26,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 15:21:27,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 15:21:27,831 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 15:21:27,832 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 21213.19921875, u'ReplayWALOffset': u'2913:6975488', u'NTablesCreated': 153288, u'RecentReadIOAmpSet': 0.018630648031830788, u'ReadIOAmp': 0.29530394077301025, u'BlockCacheMisses': 5314521, u'NFlushes': 51179, u'RecentWriteAmp': 3.725177526473999, u'NumSyncFlushes': 0, u'TotalMemUsed': 89886137, u'NBloomFilterHits': 287060085, u'Fragmentation': 0.0022416117135435343, u'NumNormalFlushes': 26312, u'RecentBlockCacheHitRatio': 72.69005584716797, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 47916, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 369688, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 183135030, u'CheckpointOverhead': 0, u'NItemsGet': 0, u'NWriteIOs': 102151, u'NReadBytes': 0, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 229616612, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 11732074496L, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 0}, {u'TargetSize': 45923322952L, u'LogicalSize': 23464148992L, u'SizeRatio': 0.5109418805891989, u'NWriteBytes': 23464148992L, u'BloomFilterMemUsed': 0, u'NumTables': 552, u'WastedSpace': 502487516, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 23464148992L}], u'NItemsGarbageCollected': 0, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 141046715, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 58272, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 0, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 171174, u'CompactWriter': {u'NDataBlockWrites': 0, u'NWriteBytes': 0, u'NIndexBlockWrites': 0, u'NBytesIncoming': 0, u'NSyncs': 0, u'NInserts': 0, u'NWriteIOs': 0}, u'NWriteBytes': 23502967725L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 23464148992L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.7587484089319728, u'BufferMemUsed': 502756, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 23464148992L, u'NTableFiles': 552, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 150536, u'TreeSnapshotMemoryUsed': 100512, u'TotalDiskUsage': 30525743104L, u'BytesIncoming': 30975969700L, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 0, u'NItemsPut': 21426350, u'NSyncs': 24867, u'CompactReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'ActiveDiskUsage': 23464148992L, u'NTablesDeleted': 0, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 7157756, u'NWriteBytes': 23502967725L, u'NIndexBlockWrites': 162867, u'NBytesIncoming': 31168562446L, u'NSyncs': 24867, u'NInserts': 21426182, u'NWriteIOs': 102151}}, u'NReadBytes': 21385639100L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 191.23333740234375, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 153066588, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 5062, u'NTxns': 3114851, u'NWriteBytes': 39639216128L, u'NSegments': 2913, u'NSyncs': 3114851, u'NLoggedBytes': 39639216128L, u'NActiveSegments': 1, u'NWrites': 3114897, u'NInserts': 47395296, u'DiskUsed': 7516160}, u'NGets': 9742505, u'NReadBytesSet': 1508780968, u'NReadBytesCompact': 11584179160L, u'IndexResidentRatio': 0.07353949546813965, u'CompactQueueSize': 0, u'NSyncsPerSec': 1.100000023841858, u'BlockCacheHitRatio': 77.49811553955078, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 9801459940L, u'BloomFilterMemUsedRatio': 0.12916189432144165, u'NWriteBytes': 76684412582L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.07872121036052704, u'WriteAmp': 2.3573191165924072, u'NBlocksCached': 23179, u'LogicalDataSize': 24201450198L, u'ActiveBloomFilterMemUsed': 19255656, u'NFileCountCompacts': 5666, u'NTableFiles': 1433, u'FlushQueueSize': 72, u'TableObjectMemUsed': 369024, u'BlockCacheQuota': 103093474, u'TotalDiskUsage': 31424991232L, u'NPurgerRuns': 44, u'ReadIOAmpSet': 0.0, u'NSyncs': 74601, u'NReadIOsSetPerSec': 4.0, u'NReadIOsPerSec': 191.8000030517578, u'NTablesDeleted': 104707, u'BloomFilterFPR': 0.0183258093893528, u'NBloomFilterHitsPerSec': 2508.566650390625, u'WriteCacheMemUsed': 748616, u'RecentBloomFilterFPR': 0.02272520586848259, u'BytesPerRead': 851.185546875, u'BytesIncomingPerSec': 406934.78125, u'NReadIOsGetPerSec': 7.633333206176758, u'NGetStats': 224902, u'TotalIndexBlocksSize': 195376948, u'NGetStatsPerSec': 10.433333396911621, u'NWriteBytesCompact': 11364696064L, u'NReadIOsSet': 525330, u'ActiveDataSize': 24201450198L, u'NWriteBytesCompactPerSec': 482781.875, u'NBloomFilterMissesPerSec': 58.33333206176758, u'NCommitBatches': 3114851, u'NGetStatsComputed': 224902, u'NWriteIOs': 3391685, u'NReadIOsGet': 3402330, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 25773368, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 214.6999969482422, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.7333333492279053, u'ManagedFileObjectMemUsed': 142848, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 67679188, u'NWriterCompacts': 0, u'NReadIOs': 7757896, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 270012357, u'NTablesCreated': 54643, u'ReadIOAmp': 0.1089764445950934, u'MemtablesMemUsed': 189464, u'NBloomFilterHits': 287060085, u'TotalIndexBlocksSize': 12231490, u'CheckpointOverhead': 16527360, u'NItemsGet': 31168855, u'NWriteIOs': 99041, u'NReadBytes': 21328688424L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 2518820, u'SizeRatio': 0.05459395321932706, u'NWriteBytes': 1740005376, u'BloomFilterMemUsed': 34736, u'NumTables': 95, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2387968}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 29892154, u'SizeRatio': 0.6478949893604625, u'NWriteBytes': 3485671424L, u'BloomFilterMemUsed': 920528, u'NumTables': 78, u'WastedSpace': 36600, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 29847552}, {u'TargetSize': 79154773, u'LogicalSize': 54372138, u'SizeRatio': 0.6869091520229614, u'NWriteBytes': 4839501824L, u'BloomFilterMemUsed': 1664264, u'NumTables': 55, u'WastedSpace': 134314, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 54476800}, {u'TargetSize': 791547796, u'LogicalSize': 791547796, u'SizeRatio': 1.0, u'NWriteBytes': 3137851392L, u'BloomFilterMemUsed': 16634664, u'NumTables': 331, u'WastedSpace': 2306104, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 793841664}], u'NItemsGarbageCollected': 0, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 12014653, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 64128, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 7650991, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 203017, u'CompactWriter': {u'NDataBlockWrites': 4117976, u'NWriteBytes': 11341651968L, u'NIndexBlockWrites': 55324, u'NBytesIncoming': 15660663456L, u'NSyncs': 0, u'NInserts': 270012357, u'NWriteIOs': 26258}, u'NWriteBytes': 13114081905L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 3396671, u'NReadBytes': 9793341767L, u'NDataBlockReads': 22054008, u'NBytesOutgoing': 0, u'NIndexBlockReads': 31724585}, u'NBloomFilterMisses': 5358813, u'NItemsMerged': 0, u'LogicalDiskSize': 791547796, u'TotalBloomFilterMemUsed': 19260352, u'WriteAmp': 12.490929421641386, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 19254192, u'LogicalDataSize': 737175658, u'NTableFiles': 668, u'NFileCountCompacts': 2853, u'TableObjectMemUsed': 165664, u'TreeSnapshotMemoryUsed': 103392, u'TotalDiskUsage': 897503232, u'BytesIncoming': 1049888400, u'NItemsDel': 0, u'PendingCheckpointBytes': 622592, u'NCompacts': 3402, u'NItemsPut': 21426350, u'NSyncs': 24867, u'CompactReader': {u'NReadIOs': 4254320, u'NReadBytes': 11535346657L, u'NDataBlockReads': 4155436, u'NBytesOutgoing': 0, u'NIndexBlockReads': 98884}, u'ActiveDiskUsage': 880553984, u'NTablesDeleted': 54084, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.018325809435202782, u'Writer': {u'NDataBlockWrites': 362908, u'NWriteBytes': 1772429937, u'NIndexBlockWrites': 47916, u'NBytesIncoming': 1242715806, u'NSyncs': 24867, u'NInserts': 21426182, u'NWriteIOs': 72783}}, u'NReadBytesCompactPerSec': 479131.65625, u'TableMetaMemUsed': 435313, u'NDeletes': 0, u'BlockCacheHitsPerSec': 509.0, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 5358813, u'LogicalDiskSize': 24255822336L, u'NSets': 21426350, u'TotalBloomFilterMemUsed': 19262016, u'BufferMemUsed': 502756, u'RecentReadAmp': 0.0, u'NGetsPerSec': 96.96666717529297, u'NInserts': 0, u'NCommitBatchesPerSec': 186.10000610351562, u'FragmentationWithOverhead': 0.2298661321401596, u'NCompactsPerSec': 0.20000000298023224, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 500344.875, u'TreeSnapshotMemoryUsed': 249184, u'BytesIncoming': 32530347073L, u'NCompacts': 6215, u'NumPersistentFlushes': 24867, u'NWriteIOsPerSec': 190.93333435058594, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 218.76795959472656, u'ActiveDiskUsage': 24345571328L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 10.433333396911621, u'NReadBytesSetPerSec': 11040.7001953125, u'BlockCacheHits': 18303594, u'NWriteBytesPerSec': 1515904.25, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 50729, u'ReadIOAmp': 0.0009073231673803613, u'MemtablesMemUsed': 189464, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 10428, u'CheckpointOverhead': 729088, u'NItemsGet': 6237028, u'NWriteIOs': 75596, u'NReadBytes': 56950676, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 860988, u'SizeRatio': 0.018661412325772373, u'NWriteBytes': 392572928, u'BloomFilterMemUsed': 760, u'NumTables': 95, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 778240}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 125548, u'SizeRatio': 0.0027211796153675427, u'NWriteBytes': 23044096, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 16623381, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 5220, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 20448, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 106905, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 61122, u'CompactWriter': {u'NDataBlockWrites': 2813, u'NWriteBytes': 23044096, u'NIndexBlockWrites': 2813, u'NBytesIncoming': 12441481, u'NSyncs': 0, u'NInserts': 149089, u'NWriteIOs': 2813}, u'NWriteBytes': 428146824, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 5659, u'NReadBytes': 8118173, u'NDataBlockReads': 3103175, u'NBytesOutgoing': 0, u'NIndexBlockReads': 3103175}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 125548, u'TotalBloomFilterMemUsed': 1664, u'WriteAmp': 0.23635017555759222, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1464, u'LogicalDataSize': 125548, u'NTableFiles': 213, u'NFileCountCompacts': 2813, u'TableObjectMemUsed': 52824, u'TreeSnapshotMemoryUsed': 45280, u'TotalDiskUsage': 1744896, u'BytesIncoming': 1811493573, u'NItemsDel': 0, u'PendingCheckpointBytes': 155648, u'NCompacts': 2813, u'NItemsPut': 16624407, u'NSyncs': 24867, u'CompactReader': {u'NReadIOs': 101246, u'NReadBytes': 48832503, u'NDataBlockReads': 50623, u'NBytesOutgoing': 0, u'NIndexBlockReads': 50623}, u'ActiveDiskUsage': 868352, u'NTablesDeleted': 50623, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 47916, u'NWriteBytes': 405102728, u'NIndexBlockWrites': 47916, u'NBytesIncoming': 34639020, u'NSyncs': 24867, u'NInserts': 438355, u'NWriteIOs': 72783}}}}} 2021-12-01 15:21:27,841 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.0022416117135435343, 0.0022941313218325377, 0.002015193458646536, 0.002324556466192007, 0.0021804943680763245, 0.002765642013400793, 0.0024411696940660477, 0.0021549160592257977, 0.0023552137427031994, 0.0020246657077223063, 0.0023817908950150013, 0.002493878360837698, 0.0025022635236382484, 0.002096407348290086, 0.002557255793362856, 0.0025922060012817383, 0.0024967764038592577, 0.0023721959441900253, 0.0024138230364769697, 0.0027135666459798813, 0.002201174385845661, 0.0026931038592010736, 0.002556746592745185, 0.002837071195244789, 0.0020925782155245543, 0.002808792982250452, 0.0026731863617897034, 0.0022596511989831924, 0.002629559952765703, 0.0022674344945698977, 0.0026121793780475855, 0.0021393971983343363, 0.0026932943146675825, 0.0023969365283846855, 0.002872421173378825, 0.0029118629172444344, 0.0025225942954421043, 0.001633833278901875, 0.0022486140951514244, 0.0025926704984158278, 0.002632267540320754, 0.0025036316365003586, 0.0024240112397819757, 0.0018561369506642222, 0.0021922013256698847, 0.0022548241540789604, 0.0023967749439179897, 0.002344548935070634, 0.002319935243576765, 0.0019873641431331635, 0.002352714305743575, 0.0029873005114495754, 0.0021521528251469135, 0.0022172534372657537, 0.0023673675023019314, 0.0017370587447658181, 0.0019436674192547798, 0.0022926563397049904, 0.0023635465186089277, 0.0017820706125348806, 0.002538592554628849, 0.0018720794469118118, 0.002089310670271516, 0.002391811925917864], '172.23.110.66': [0.0024939149152487516, 0.0021007375326007605, 0.002611503703519702, 0.0021880005951970816, 0.002742948243394494, 0.002491206396371126, 0.002252149861305952, 0.0022760566789656878, 0.002282000845298171, 0.0021774868946522474, 0.0027259541675448418, 0.002180114621296525, 0.0026868656277656555, 0.0024218971375375986, 0.0022028167732059956, 0.002307652961462736, 0.00247765495441854, 0.002475747372955084, 0.002730895997956395, 0.0026096301153302193, 0.0027981132734566927, 0.0025442405603826046, 0.0026306987274438143, 0.002329097827896476, 0.00251530553214252, 0.0023596358951181173, 0.0023948841262608767, 0.002318717073649168, 0.0021518648136407137, 0.0024667561519891024, 0.0016075385501608253, 0.0022604777477681637, 0.0024134665727615356, 0.0028411527164280415, 0.0022247275337576866, 0.0024975701235234737, 0.0025673522613942623, 0.0018893826054409146, 0.0024243383668363094, 0.0022212876938283443, 0.0023614924866706133, 0.002471758285537362, 0.002455576555803418, 0.0028461795300245285, 0.002510031685233116, 0.0020611744839698076, 0.002206519478932023, 0.00224901526235044, 0.002127568470314145, 0.0024237323086708784, 0.0023756232112646103, 0.002508356934413314, 0.002504774136468768, 0.002007761737331748, 0.0025947229005396366, 0.00267780595459044, 0.00244826078414917, 0.00258748815394938, 0.002612530020996928, 0.0025073832366615534, 0.002400142140686512, 0.002614497672766447, 0.0021753450855612755, 0.0020475347992032766], '172.23.110.64': [0.0022234146017581224, 0.002527899108827114, 0.0020567569881677628, 0.0024150044191628695, 0.0025121888611465693, 0.0025077685713768005, 0.002370118396356702, 0.001972913509234786, 0.00241631967946887, 0.0024957263376563787, 0.0022888134699314833, 0.0023345232475548983, 0.0020164253655821085, 0.002202080562710762, 0.00229130988009274, 0.001682913163676858, 0.0025677455123513937, 0.0025699862744659185, 0.0018582910997793078, 0.002481174888089299, 0.0016975300386548042, 0.00261866906657815, 0.0024680288042873144, 0.0025963953230530024, 0.0020359940826892853, 0.002601386746391654, 0.0021907964255660772, 0.0021182314958423376, 0.002775561762973666, 0.002599793951958418, 0.002161510055884719, 0.0027297334745526314, 0.0027865346055477858, 0.0018869200721383095, 0.002287589479237795, 0.002590629504993558, 0.0022109798155725002, 0.0017330119153484702, 0.002267823787406087, 0.002528937766328454, 0.0020939791575074196, 0.0026525924913585186, 0.0028460004832595587, 0.0026993220672011375, 0.002425026148557663, 0.0020654022227972746, 0.0027526472695171833, 0.0017714494606480002, 0.0023511487524956465, 0.0024831565096974373, 0.0024050248321145773, 0.0024220224004238844, 0.0025446289218962193, 0.0027132942341268063, 0.0027059807907789946, 0.0018588807433843613, 0.002387530170381069, 0.001955490792170167, 0.002727484330534935, 0.0023606563918292522, 0.0024163336493074894, 0.0020715356804430485, 0.0025543973315507174, 0.0020314499270170927]} 2021-12-01 15:21:28,180 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 1.0 2021-12-01 15:21:28,522 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 1.0 2021-12-01 15:21:28,868 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 1.0 2021-12-01 15:21:28,868 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 1.0, '172.23.110.66': 1.0, '172.23.110.64': 1.0} 2021-12-01 15:21:28,871 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 15:21:28,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:21:29,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 15:21:29,357 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:21:29,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 15:21:29,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 15:21:29,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 15:21:29,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 15:21:29,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 15:21:29,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 15:21:29,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 15:21:29,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 15:21:29,539 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-01 15:21:29,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 15:21:29,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 15:21:29,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 15:21:29,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 15:21:29,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 15:21:29,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 15:21:29,803 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-01 15:21:29,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 15:21:29,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 15:21:29,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 15:21:29,848 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-01 15:21:29,848 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 15:21:29,851 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 15:21:29,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:21:30,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 15:21:30,322 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:21:30,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 15:21:30,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 15:21:30,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 15:21:30,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 15:21:30,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 15:21:30,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 15:21:30,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 15:21:30,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 15:21:30,476 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-01 15:21:30,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 15:21:30,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 15:21:30,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 15:21:30,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 15:21:30,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 15:21:30,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 15:21:30,710 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-01 15:21:30,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 15:21:30,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 15:21:30,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 15:21:30,762 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-01 15:21:30,763 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 15:21:30,766 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 15:21:30,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 15:21:31,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 15:21:31,273 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 15:21:31,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 15:21:31,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 15:21:31,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 15:21:31,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 15:21:31,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 15:21:31,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 15:21:31,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 15:21:31,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 15:21:31,430 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-01 15:21:31,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 15:21:31,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 15:21:31,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 15:21:31,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 15:21:31,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 15:21:31,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 15:21:31,661 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-01 15:21:31,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 15:21:31,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 15:21:31,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 15:21:31,726 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-01 15:21:31,727 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 ############################################################ # # Step 2.1: Update 40000000 RandonKey keys to create 50 percent fragmentation # ############################################################ ############################################################ # # RAM FootPrint: 6882.12890625 # ############################################################ Read Start: 0 Read End: 0 Update Start: 0 Update End: 40000000 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 0 Create End: 0 Final Start: 0 Final End: 40000000 2021-12-01 15:21:31,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:31,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1638400892.8 2021-12-01 15:21:32,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:32,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1638400893.81 2021-12-01 15:21:33,821 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:33,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1638400894.82 2021-12-01 15:21:34,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:34,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1638400895.84 2021-12-01 15:21:35,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:35,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1638400896.85 2021-12-01 15:21:36,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:36,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1638400897.92 2021-12-01 15:21:37,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:37,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1638400898.94 2021-12-01 15:21:38,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:38,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1638400899.95 2021-12-01 15:21:39,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:39,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1638400900.96 2021-12-01 15:21:40,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:40,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1638400901.98 2021-12-01 15:21:41,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:41,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:43,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:43,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:44,022 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:44,022 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:45,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:45,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:46,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:46,073 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:47,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:47,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:48,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:48,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:49,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:49,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:50,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:50,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:51,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:51,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:52,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:52,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:53,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:53,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:54,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:54,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:55,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:55,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:56,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:56,272 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:57,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:57,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:58,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:58,301 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:21:59,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:21:59,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:00,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:00,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:01,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:01,413 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:02,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:02,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:03,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:03,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:04,451 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:04,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:05,464 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:05,466 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:06,479 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:06,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:07,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:07,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:08,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:08,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:09,530 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:09,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:10,548 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:10,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:11,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:11,561 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:12,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:12,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:13,584 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:13,584 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:14,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:14,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:15,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:15,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:16,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:16,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:17,637 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:17,637 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:18,650 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:18,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:19,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:19,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:20,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:20,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 15:22:21,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 15:22:21,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection49_5_1638400897.92 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection35_0_1638400902.99 Loader_GleamBookUsers0__default_VolumeCollection47_9_1638400893.81 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638400904.01 Loader_GleamBookUsers0__default_VolumeCollection44_8_1638400894.82 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638400905.02 Loader_GleamBookUsers0__default_VolumeCollection42_4_1638400898.94 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638400906.06 Loader_GleamBookUsers0__default_VolumeCollection40_2_1638400900.96 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638400907.08 Loader_GleamBookUsers0__default_VolumeCollection41_1_1638400901.98 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638400908.09 Loader_GleamBookUsers0__default_VolumeCollection46_10_1638400892.8 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638400909.11 Loader_GleamBookUsers0__default_VolumeCollection45_7_1638400895.84 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638400910.13 Loader_GleamBookUsers0__default_VolumeCollection48_6_1638400896.85 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638400911.14 Loader_GleamBookUsers0__default_VolumeCollection43_3_1638400899.95 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638400912.2 Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638400906.06 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638400913.21 Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638400904.01 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638400914.23 Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638400908.09 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638400915.24 Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638400912.2 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638400916.26 Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638400905.02 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638400917.27 Loader_GleamBookUsers0__default_VolumeCollection35_0_1638400902.99 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638400918.29 Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638400907.08 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638400919.37 Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638400909.11 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638400920.39 Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638400911.14 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638400921.4 Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638400910.13 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638400922.41 Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638400914.23 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638400923.43 Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638400916.26 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638400924.44 Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638400918.29 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638400925.45 Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638400920.39 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638400926.47 Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638400913.21 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638400927.49 Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638400915.24 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638400928.5 Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638400917.27 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638400929.52 Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638400919.37 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638400930.53 Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638400921.4 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638400931.55 Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638400924.44 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638400932.56 Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638400922.41 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638400933.57 Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638400926.47 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638400934.59 Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638400923.43 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638400935.6 Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638400928.5 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638400936.61 Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638400925.45 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638400937.62 Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638400930.53 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638400938.64 Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638400927.49 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638400939.65 Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638400932.56 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638400940.66 Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638400929.52 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638400941.68 Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638400934.59 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638400942.69 Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638400933.57 is completed! Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638400931.55 is completed! Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638400936.61 is completed! Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638400935.6 is completed! Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638400938.64 is completed! Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638400940.66 is completed! Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638400937.62 is completed! Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638400939.65 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638400942.69 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638400941.68 is completed! 2021-12-01 19:20:58,312 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-01 19:20:58,316 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 19:20:58,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:20:58,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 19:20:58,802 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:20:58,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 19:20:58,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 19:20:58,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 19:20:58,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 19:20:58,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 19:20:58,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 19:20:58,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 19:20:58,971 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 19:20:58,973 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415258.97 2021-12-01 19:20:58,973 | infra | INFO | pool-3-thread-11 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415258.97' started 2021-12-01 19:20:58,976 | infra | DEBUG | pool-3-thread-11 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-01 19:20:58,979 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 19:20:59,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:20:59,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 19:20:59,443 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:20:59,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 19:20:59,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 19:20:59,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 19:20:59,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 19:20:59,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 19:20:59,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 19:20:59,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 19:20:59,562 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 19:20:59,566 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415259.56 2021-12-01 19:20:59,568 | infra | INFO | pool-3-thread-12 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415259.56' started 2021-12-01 19:20:59,569 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-01 19:20:59,575 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 19:20:59,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:20:59,766 | test | DEBUG | pool-3-thread-11 | [task:_compare:2725] Comparing 0 == 0 2021-12-01 19:20:59,766 | test | DEBUG | pool-3-thread-11 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.64': 0} for bucket 'GleamBookUsers0' 2021-12-01 19:20:59,766 | infra | INFO | pool-3-thread-11 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415258.97' completed 2021-12-01 19:20:59,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 19:21:00,072 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:21:00,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 19:21:00,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 19:21:00,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 19:21:00,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 19:21:00,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 19:21:00,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 19:21:00,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 19:21:00,206 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 19:21:00,207 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415260.21 2021-12-01 19:21:00,207 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415258.97 2021-12-01 19:21:00,207 | infra | INFO | pool-3-thread-13 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415260.21' started 2021-12-01 19:21:00,210 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 19:21:00,210 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-01 19:21:00,210 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415259.56 2021-12-01 19:21:00,354 | test | DEBUG | pool-3-thread-12 | [task:_compare:2725] Comparing 0 == 0 2021-12-01 19:21:00,355 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.65': 0} for bucket 'GleamBookUsers0' 2021-12-01 19:21:00,355 | infra | INFO | pool-3-thread-12 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415259.56' completed 2021-12-01 19:21:00,355 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 19:21:00,355 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415260.21 2021-12-01 19:21:01,042 | test | DEBUG | pool-3-thread-13 | [task:_compare:2725] Comparing 0 == 0 2021-12-01 19:21:01,042 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.66': 0} for bucket 'GleamBookUsers0' 2021-12-01 19:21:01,042 | infra | INFO | pool-3-thread-13 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638415260.21' completed 2021-12-01 19:21:01,043 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 19:21:01,082 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3946] Stats: {u'opsPerSec': 21826.3, u'diskFetches': 0, u'dataUsed': 3768630891642L, u'itemCount': 2000000000, u'quotaPercentUsed': 82.16366338481505, u'storageTotals': {u'hdd': {u'usedByData': 6785870952984L, u'used': 6940165511574L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 29035644479082L}, u'ram': {u'usedByData': 264667685368L, u'used': 354878623744L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 6785870952984L, u'vbActiveNumNonResident': 1862849343, u'memUsed': 264667685368L} 2021-12-01 19:21:01,433 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 2000000000 | 300.00 GiB | 246.49 GiB | 6.17 TiB | 6.86019555 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ 2021-12-01 19:21:01,434 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+---------------+---------------+---------------+ | 2000000000 | 2000000000 | 0-40000000 | 0-0 | 0-0 | 0-0 | +---------------+---------------+---------------+---------------+---------------+---------------+ 2021-12-01 19:21:01,798 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 6.86019555 2021-12-01 19:21:01,801 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 19:21:01,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:21:02,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 19:21:02,305 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:21:02,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 19:21:02,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 19:21:02,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 19:21:02,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 19:21:02,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 19:21:02,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 19:21:02,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 19:21:02,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 19:21:03,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 19:21:03,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 19:21:03,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 19:21:03,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 19:21:05,259 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 19:21:05,263 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 19:21:05,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:21:05,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 19:21:05,737 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:21:05,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 19:21:05,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 19:21:05,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 19:21:05,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 19:21:05,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 19:21:05,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 19:21:05,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 19:21:05,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 19:21:06,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 19:21:06,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 19:21:06,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 19:21:06,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 19:21:08,480 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 19:21:08,483 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 19:21:08,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:21:08,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 19:21:08,970 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:21:08,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 19:21:08,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 19:21:09,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 19:21:09,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 19:21:09,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 19:21:09,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 19:21:09,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 19:21:09,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 19:21:09,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 19:21:09,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 19:21:09,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 19:21:09,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 19:21:11,618 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 19:21:11,619 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 6485429MB 2021-12-01 19:21:11,621 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 8650MB 2021-12-01 19:21:11,621 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 192015MB 2021-12-01 19:21:11,621 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 6293018MB 2021-12-01 19:21:11,624 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 19:21:11,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:21:12,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 19:21:12,132 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:21:12,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 19:21:12,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 19:21:12,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 19:21:12,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 19:21:12,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 19:21:12,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 19:21:12,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 19:21:12,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 19:21:12,357 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-01 19:21:12,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 19:21:12,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 19:21:13,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 19:21:13,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 19:21:14,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 19:21:14,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 19:21:14,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 19:21:15,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 19:21:15,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 19:21:16,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 19:21:16,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 19:21:17,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 19:21:17,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 19:21:17,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 19:21:18,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 19:21:18,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 19:21:18,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 19:21:19,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 19:21:19,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 19:21:20,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 19:21:20,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 19:21:21,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 19:21:21,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 19:21:22,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 19:21:22,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 19:21:23,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 19:21:23,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 19:21:24,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 19:21:24,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 19:21:25,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 19:21:25,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 19:21:26,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 19:21:26,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 19:21:27,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 19:21:27,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 19:21:27,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 19:21:28,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 19:21:28,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 19:21:29,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 19:21:29,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 19:21:30,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 19:21:30,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 19:21:30,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 19:21:31,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 19:21:31,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 19:21:32,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 19:21:32,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 19:21:33,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 19:21:33,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 19:21:33,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 19:21:34,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 19:21:34,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 19:21:34,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 19:21:35,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 19:21:35,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 19:21:36,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 19:21:36,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 19:21:37,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 19:21:37,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 19:21:37,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 19:21:38,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 19:21:38,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 19:21:39,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 19:21:39,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 19:21:39,785 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 19:21:39,789 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 19:21:39,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:21:40,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 19:21:40,265 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:21:40,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 19:21:40,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 19:21:40,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 19:21:40,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 19:21:40,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 19:21:40,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 19:21:40,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 19:21:40,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 19:21:40,464 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-01 19:21:40,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 19:21:40,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 19:21:41,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 19:21:41,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 19:21:42,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 19:21:42,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 19:21:43,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 19:21:43,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 19:21:44,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 19:21:44,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 19:21:45,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 19:21:45,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 19:21:46,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 19:21:46,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 19:21:46,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 19:21:47,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 19:21:47,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 19:21:48,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 19:21:48,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 19:21:49,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 19:21:49,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 19:21:50,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 19:21:50,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 19:21:51,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 19:21:51,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 19:21:52,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 19:21:52,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 19:21:52,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 19:21:53,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 19:21:53,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 19:21:53,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 19:21:54,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 19:21:54,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 19:21:55,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 19:21:55,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 19:21:56,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 19:21:56,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 19:21:57,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 19:21:57,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 19:21:58,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 19:21:58,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 19:21:58,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 19:21:59,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 19:21:59,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 19:22:00,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 19:22:00,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 19:22:01,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 19:22:01,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 19:22:02,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 19:22:02,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 19:22:02,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 19:22:03,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 19:22:03,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 19:22:04,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 19:22:04,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 19:22:05,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 19:22:05,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 19:22:06,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 19:22:06,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 19:22:06,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 19:22:07,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 19:22:07,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 19:22:08,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 19:22:08,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 19:22:08,907 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 19:22:08,911 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 19:22:09,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:22:09,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 19:22:09,375 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:22:09,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 19:22:09,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 19:22:09,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 19:22:09,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 19:22:09,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 19:22:09,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 19:22:09,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 19:22:09,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 19:22:09,602 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-01 19:22:09,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 19:22:10,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 19:22:10,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 19:22:11,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 19:22:11,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 19:22:12,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 19:22:12,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 19:22:12,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 19:22:13,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 19:22:13,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 19:22:14,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 19:22:14,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 19:22:15,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 19:22:15,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 19:22:16,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 19:22:16,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 19:22:17,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 19:22:17,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 19:22:17,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 19:22:18,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 19:22:18,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 19:22:19,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 19:22:19,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 19:22:20,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 19:22:20,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 19:22:21,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 19:22:21,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 19:22:21,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 19:22:22,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 19:22:22,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 19:22:23,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 19:22:23,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 19:22:24,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 19:22:24,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 19:22:25,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 19:22:25,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 19:22:26,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 19:22:26,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 19:22:27,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 19:22:27,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 19:22:28,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 19:22:28,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 19:22:29,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 19:22:29,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 19:22:30,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 19:22:30,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 19:22:31,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 19:22:31,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 19:22:32,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 19:22:32,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 19:22:33,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 19:22:33,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 19:22:34,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 19:22:34,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 19:22:34,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 19:22:35,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 19:22:35,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 19:22:36,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 19:22:36,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 19:22:37,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 19:22:37,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 19:22:37,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 19:22:38,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 19:22:38,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 19:22:39,477 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 19:22:39,479 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 47906.734375, u'ReplayWALOffset': u'5248:5816320', u'NTablesCreated': 419931, u'RecentReadIOAmpSet': 0.03576575219631195, u'ReadIOAmp': 520.5797119140625, u'BlockCacheMisses': 22904230, u'NFlushes': 91539, u'RecentWriteAmp': 3.2770161628723145, u'NumSyncFlushes': 3, u'TotalMemUsed': 131479238, u'NBloomFilterHits': 481165759, u'Fragmentation': 0.43051257729530334, u'NumNormalFlushes': 47905, u'RecentBlockCacheHitRatio': 87.56389617919922, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 228680, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 8217624, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 261325203, u'CheckpointOverhead': 8101888, u'NItemsGet': 0, u'NWriteIOs': 331171, u'NReadBytes': 17770993519L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 15255489, u'SizeRatio': 0.33065381917086517, u'NWriteBytes': 730640384, u'BloomFilterMemUsed': 0, u'NumTables': 44, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 569344}, {u'TargetSize': 46137344, u'LogicalSize': 28456156, u'SizeRatio': 0.6167705709284003, u'NWriteBytes': 580325376, u'BloomFilterMemUsed': 0, u'NumTables': 19, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 372736}, {u'TargetSize': 340547964, u'LogicalSize': 282076328, u'SizeRatio': 0.8283013196931049, u'NWriteBytes': 1717956608, u'BloomFilterMemUsed': 0, u'NumTables': 144, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 3395584}, {u'TargetSize': 17375707354L, u'LogicalSize': 15265128576L, u'SizeRatio': 0.8785327851695125, u'NWriteBytes': 3577847808L, u'BloomFilterMemUsed': 0, u'NumTables': 423, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 150401024}, {u'TargetSize': 68109593842L, u'LogicalSize': 34751414709L, u'SizeRatio': 0.5102278952010199, u'NWriteBytes': 47278526464L, u'BloomFilterMemUsed': 0, u'NumTables': 179, u'WastedSpace': 696617788, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 34751414272L}], u'NItemsGarbageCollected': 11061212, u'NInternalFragmentationCompacts': 33, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 210690155, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 105792, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 6041725, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 310764, u'CompactWriter': {u'NDataBlockWrites': 2089270, u'NWriteBytes': 6157361152L, u'NIndexBlockWrites': 111537, u'NBytesIncoming': 8511254363L, u'NSyncs': 0, u'NInserts': 360366763, u'NWriteIOs': 105309}, u'NWriteBytes': 53837296895L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 11061212, u'LogicalDiskSize': 34751414709L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.8651284007004029, u'BufferMemUsed': 494628, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 19486286133L, u'NTableFiles': 1047, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 273296, u'TreeSnapshotMemoryUsed': 126832, u'TotalDiskUsage': 43234770944L, u'BytesIncoming': 62230412100L, u'NItemsDel': 0, u'PendingCheckpointBytes': 10399744, u'NCompacts': 31523, u'NItemsPut': 64277950, u'NSyncs': 43634, u'CompactReader': {u'NReadIOs': 6041725, u'NReadBytes': 17770993519L, u'NDataBlockReads': 5866300, u'NBytesOutgoing': 0, u'NIndexBlockReads': 175425}, u'ActiveDiskUsage': 34906152960L, u'NTablesDeleted': 142454, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 14448562, u'NWriteBytes': 47679935743L, u'NIndexBlockWrites': 358361, u'NBytesIncoming': 62801585005L, u'NSyncs': 43634, u'NInserts': 64268024, u'NWriteIOs': 225862}}, u'NReadBytes': 13640082593033L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 155.6999969482422, u'NInternalFragmentationCompacts': 33, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 224081358, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 5077, u'NTxns': 4470174, u'NWriteBytes': 75252957184L, u'NSegments': 5249, u'NSyncs': 4470174, u'NLoggedBytes': 75252957184L, u'NActiveSegments': 2, u'NWrites': 4483983, u'NInserts': 82593568, u'DiskUsed': 23699456}, u'NGets': 9742505, u'NReadBytesSet': 30877069083L, u'NReadBytesCompact': 46036803974L, u'IndexResidentRatio': 0.09143517166376114, u'CompactQueueSize': 0, u'NSyncsPerSec': 1.100000023841858, u'BlockCacheHitRatio': 77.49496459960938, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 13594045789059L, u'BloomFilterMemUsedRatio': 0.14129073917865753, u'NWriteBytes': 160451486120L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.0, u'WriteAmp': 2.5035603046417236, u'NBlocksCached': 33831, u'LogicalDataSize': 20287093638L, u'ActiveBloomFilterMemUsed': 20867944, u'NFileCountCompacts': 10143, u'NTableFiles': 1829, u'FlushQueueSize': 72, u'TableObjectMemUsed': 467232, u'BlockCacheQuota': 101565710, u'TotalDiskUsage': 44239785984L, u'NPurgerRuns': 88, u'ReadIOAmpSet': 0.0, u'NSyncs': 130902, u'NReadIOsSetPerSec': 9.100000381469727, u'NReadIOsPerSec': 184.1999969482422, u'NTablesDeleted': 333207, u'BloomFilterFPR': 0.015378240495920181, u'NBloomFilterHitsPerSec': 1782.8333740234375, u'WriteCacheMemUsed': 10857544, u'RecentBloomFilterFPR': 0.015281230211257935, u'BytesPerRead': 1392164.5, u'BytesIncomingPerSec': 440113.0, u'NReadIOsGetPerSec': 17.96666717529297, u'NGetStats': 339347, u'TotalIndexBlocksSize': 275045697, u'NGetStatsPerSec': 4.433333396911621, u'NWriteBytesCompact': 33330405376L, u'NReadIOsSet': 11643914, u'ActiveDataSize': 20287093638L, u'NWriteBytesCompactPerSec': 429397.34375, u'NBloomFilterMissesPerSec': 27.66666603088379, u'NCommitBatches': 4470174, u'NGetStatsComputed': 339347, u'NWriteIOs': 5140627, u'NReadIOsGet': 5083394474L, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 25391428, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 254.43333435058594, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.7333333492279053, u'ManagedFileObjectMemUsed': 180864, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 97104825, u'NWriterCompacts': 0, u'NReadIOs': 5100082462L, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 649995140, u'NTablesCreated': 100137, u'ReadIOAmp': 96.65209919144826, u'MemtablesMemUsed': 549912, u'NBloomFilterHits': 481165759, u'TotalIndexBlocksSize': 13708062, u'CheckpointOverhead': 14917632, u'NItemsGet': 52594655, u'NWriteIOs': 190725, u'NReadBytes': 13622206328686L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 1998262, u'SizeRatio': 0.04331116242842241, u'NWriteBytes': 3400388608L, u'BloomFilterMemUsed': 28088, u'NumTables': 73, u'WastedSpace': 161, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1904640}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 30332386, u'SizeRatio': 0.6574367609891024, u'NWriteBytes': 10886287360L, u'BloomFilterMemUsed': 927456, u'NumTables': 55, u'WastedSpace': 64598, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 30347264}, {u'TargetSize': 87188765, u'LogicalSize': 71205551, u'SizeRatio': 0.8166826425400108, u'NWriteBytes': 7347896320L, u'BloomFilterMemUsed': 2180928, u'NumTables': 38, u'WastedSpace': 192450, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 71389184}, {u'TargetSize': 871887702, u'LogicalSize': 871887702, u'SizeRatio': 1.0, u'NWriteBytes': 9190649856L, u'BloomFilterMemUsed': 17730064, u'NumTables': 233, u'WastedSpace': 2875084, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 874749952}], u'NItemsGarbageCollected': 18932749, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 13386319, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 50688, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 5093848045L, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 160457, u'CompactWriter': {u'NDataBlockWrites': 9900458, u'NWriteBytes': 27131617280L, u'NIndexBlockWrites': 129506, u'NBytesIncoming': 37699651870L, u'NSyncs': 0, u'NInserts': 649995140, u'NWriteIOs': 60933}, u'NWriteBytes': 30592625205L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 5083383812L, u'NReadBytes': 13594030508437L, u'NDataBlockReads': 11512729332L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 22305008044L}, u'NBloomFilterMisses': 7515051, u'NItemsMerged': 0, u'LogicalDiskSize': 871887702, u'TotalBloomFilterMemUsed': 21069392, u'WriteAmp': 14.569633205835773, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 20866536, u'LogicalDataSize': 800682151, u'NTableFiles': 528, u'NFileCountCompacts': 5086, u'TableObjectMemUsed': 130944, u'TreeSnapshotMemoryUsed': 90384, u'TotalDiskUsage': 1002934272, u'BytesIncoming': 2099752600, u'NItemsDel': 0, u'PendingCheckpointBytes': 23052288, u'NCompacts': 6530, u'NItemsPut': 42852150, u'NSyncs': 43634, u'CompactReader': {u'NReadIOs': 10464233, u'NReadBytes': 28175820249L, u'NDataBlockReads': 10252844, u'NBytesOutgoing': 0, u'NIndexBlockReads': 211389}, u'ActiveDiskUsage': 978391040, u'NTablesDeleted': 99738, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.015378240451062525, u'Writer': {u'NDataBlockWrites': 715447, u'NWriteBytes': 3461007925L, u'NIndexBlockWrites': 86392, u'NBytesIncoming': 2485134096L, u'NSyncs': 43634, u'NInserts': 42847187, u'NWriteIOs': 129792}}, u'NReadBytesCompactPerSec': 425007.4375, u'TableMetaMemUsed': 544105, u'NDeletes': 0, u'BlockCacheHitsPerSec': 1096.300048828125, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 7515051, u'LogicalDiskSize': 35623427765L, u'NSets': 42852150, u'TotalBloomFilterMemUsed': 21070800, u'BufferMemUsed': 494628, u'RecentReadAmp': 0.0, u'NGetsPerSec': 0.0, u'NInserts': 0, u'NCommitBatchesPerSec': 140.5, u'FragmentationWithOverhead': 0.541428804397583, u'NCompactsPerSec': 0.36666667461395264, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 472914.15625, u'TreeSnapshotMemoryUsed': 262048, u'BytesIncoming': 64089325577L, u'NCompacts': 43110, u'NumPersistentFlushes': 43631, u'NWriteIOsPerSec': 146.73333740234375, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 0.0, u'ActiveDiskUsage': 35885355008L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 4.433333396911621, u'NReadBytesSetPerSec': 25057.400390625, u'BlockCacheHits': 78869595, u'NWriteBytesPerSec': 1442257.5, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 91114, u'ReadIOAmp': 0.0011907620452695073, u'MemtablesMemUsed': 2090008, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 12432, u'CheckpointOverhead': 1269760, u'NItemsGet': 8953930, u'NWriteIOs': 134748, u'NReadBytes': 105270828, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 800610, u'SizeRatio': 0.01735275441950018, u'NWriteBytes': 705024000, u'BloomFilterMemUsed': 704, u'NumTables': 88, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 720896}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 125354, u'SizeRatio': 0.0027169747786088424, u'NWriteBytes': 41426944, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 26323356, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 4884, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 24384, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 192692, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 72884, u'CompactWriter': {u'NDataBlockWrites': 5057, u'NWriteBytes': 41426944, u'NIndexBlockWrites': 5057, u'NBytesIncoming': 22725986, u'NSyncs': 0, u'NInserts': 268021, u'NWriteIOs': 5057}, u'NWriteBytes': 768606836, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 10662, u'NReadBytes': 15280622, u'NDataBlockReads': 4458555, u'NBytesOutgoing': 0, u'NIndexBlockReads': 4458555}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 125354, u'TotalBloomFilterMemUsed': 1408, u'WriteAmp': 0.2898571392134617, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1408, u'LogicalDataSize': 125354, u'NTableFiles': 254, u'NFileCountCompacts': 5057, u'TableObjectMemUsed': 62992, u'TreeSnapshotMemoryUsed': 44832, u'TotalDiskUsage': 2080768, u'BytesIncoming': 2651674677L, u'NItemsDel': 0, u'PendingCheckpointBytes': 8192, u'NCompacts': 5057, u'NItemsPut': 26330910, u'NSyncs': 43634, u'CompactReader': {u'NReadIOs': 182030, u'NReadBytes': 89990206, u'NDataBlockReads': 91015, u'NBytesOutgoing': 0, u'NIndexBlockReads': 91015}, u'ActiveDiskUsage': 811008, u'NTablesDeleted': 91015, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 86057, u'NWriteBytes': 727179892, u'NIndexBlockWrites': 86057, u'NBytesIncoming': 64035988, u'NSyncs': 43634, u'NInserts': 840012, u'NWriteIOs': 129691}}}}} 2021-12-01 19:22:39,483 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.43051257729530334, 0.43357327580451965, 0.43613389134407043, 0.43575453758239746, 0.4345661699771881, 0.4387292265892029, 0.4372095763683319, 0.4370124936103821, 0.4363447427749634, 0.44276300072669983, 0.43908655643463135, 0.4404274821281433, 0.4369151294231415, 0.43762993812561035, 0.4367634057998657, 0.43644022941589355, 0.4365817904472351, 0.43936580419540405, 0.4381030797958374, 0.43975916504859924, 0.4435546100139618, 0.4348592758178711, 0.4412572681903839, 0.4429730176925659, 0.4431270658969879, 0.4389512836933136, 0.4368695914745331, 0.4399797320365906, 0.4333602786064148, 0.43938684463500977, 0.4379119277000427, 0.4402140974998474, 0.4381754696369171, 0.4393698573112488, 0.44125017523765564, 0.438884973526001, 0.4328172504901886, 0.4357966482639313, 0.4359534978866577, 0.43418270349502563, 0.44299668073654175, 0.43378716707229614, 0.442330002784729, 0.4391922354698181, 0.4405982494354248, 0.4362667202949524, 0.4307308495044708, 0.43473389744758606, 0.4418933391571045, 0.43144240975379944, 0.43222975730895996, 0.4413243532180786, 0.43988099694252014, 0.4283526837825775, 0.43863195180892944, 0.43971705436706543, 0.4427528977394104, 0.43757179379463196, 0.43876951932907104, 0.44023430347442627, 0.43525466322898865, 0.44101202487945557, 0.4313848912715912, 0.4373176395893097], '172.23.110.66': [0.44133260846138, 0.4383646249771118, 0.437911719083786, 0.4341670870780945, 0.4398013651371002, 0.44525137543678284, 0.43796584010124207, 0.4341973066329956, 0.43765997886657715, 0.4392167329788208, 0.4394957721233368, 0.43842244148254395, 0.44327348470687866, 0.4317106604576111, 0.4414966404438019, 0.4377022683620453, 0.43748119473457336, 0.43885907530784607, 0.4423665702342987, 0.44499701261520386, 0.4382037818431854, 0.44266369938850403, 0.4407753050327301, 0.4421333074569702, 0.44189757108688354, 0.4367196559906006, 0.43921327590942383, 0.44119828939437866, 0.4354446828365326, 0.44128289818763733, 0.44319695234298706, 0.44150862097740173, 0.43727293610572815, 0.43903034925460815, 0.4416823387145996, 0.43441319465637207, 0.44053012132644653, 0.43915995955467224, 0.43933239579200745, 0.43686383962631226, 0.43805739283561707, 0.43793246150016785, 0.44866296648979187, 0.43387219309806824, 0.4371073544025421, 0.4381583333015442, 0.4375433921813965, 0.4348268210887909, 0.43821296095848083, 0.436965674161911, 0.43966948986053467, 0.4378790259361267, 0.4390581250190735, 0.43316709995269775, 0.4471888542175293, 0.43844708800315857, 0.43560728430747986, 0.44247400760650635, 0.4433111548423767, 0.43429574370384216, 0.43764278292655945, 0.43802982568740845, 0.4360767900943756, 0.443398654460907], '172.23.110.64': [0.4425680637359619, 0.4384634792804718, 0.4387998878955841, 0.4381934702396393, 0.43773338198661804, 0.43735432624816895, 0.43807944655418396, 0.43316420912742615, 0.4343743622303009, 0.4410208463668823, 0.43531927466392517, 0.43669748306274414, 0.4354046583175659, 0.4395199418067932, 0.44063326716423035, 0.4382663369178772, 0.44009634852409363, 0.43505358695983887, 0.4335850179195404, 0.4421083629131317, 0.4333435893058777, 0.4335172474384308, 0.4299597144126892, 0.44161802530288696, 0.4397576153278351, 0.43945690989494324, 0.4297201335430145, 0.43620970845222473, 0.43756574392318726, 0.43667304515838623, 0.4355957806110382, 0.43747928738594055, 0.43286100029945374, 0.4392947554588318, 0.4400483965873718, 0.43603432178497314, 0.4382724165916443, 0.43524783849716187, 0.44096076488494873, 0.4423859417438507, 0.4382009208202362, 0.4351942837238312, 0.4413354992866516, 0.43847641348838806, 0.4403814375400543, 0.4427471458911896, 0.4446125030517578, 0.4393671154975891, 0.436768501996994, 0.43676039576530457, 0.4368792176246643, 0.4372175633907318, 0.43858471512794495, 0.43893638253211975, 0.43542805314064026, 0.4354481101036072, 0.43655046820640564, 0.43896380066871643, 0.4320640563964844, 0.44015705585479736, 0.4408159554004669, 0.4325871765613556, 0.44186997413635254, 0.4369569420814514]} 2021-12-01 19:22:39,851 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 44.0 2021-12-01 19:22:40,180 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 44.0 2021-12-01 19:22:40,509 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 44.0 2021-12-01 19:22:40,509 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 44.0, '172.23.110.66': 44.0, '172.23.110.64': 44.0} 2021-12-01 19:22:40,513 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 19:22:40,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:22:40,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 19:22:40,994 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:22:40,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 19:22:41,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 19:22:41,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 19:22:41,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 19:22:41,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 19:22:41,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 19:22:41,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 19:22:41,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 19:22:41,197 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-01 19:22:41,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 19:22:41,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:41,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 19:22:41,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 19:22:41,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 19:22:41,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:41,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 19:22:41,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 19:22:41,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 19:22:41,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 19:22:41,538 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-01 19:22:41,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 19:22:41,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:41,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 19:22:41,579 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-01 19:22:41,579 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 19:22:41,584 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 19:22:41,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:22:41,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 19:22:42,055 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:22:42,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 19:22:42,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 19:22:42,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 19:22:42,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 19:22:42,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 19:22:42,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 19:22:42,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 19:22:42,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 19:22:42,196 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-01 19:22:42,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 19:22:42,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:42,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 19:22:42,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 19:22:42,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 19:22:42,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:42,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 19:22:42,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 19:22:42,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 19:22:42,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 19:22:42,545 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-01 19:22:42,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 19:22:42,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:42,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 19:22:42,589 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-01 19:22:42,591 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 19:22:42,594 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 19:22:42,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 19:22:43,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 19:22:43,101 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 19:22:43,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 19:22:43,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 19:22:43,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 19:22:43,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 19:22:43,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 19:22:43,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 19:22:43,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 19:22:43,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 19:22:43,256 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-01 19:22:43,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 19:22:43,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:43,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 19:22:43,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 19:22:43,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 19:22:43,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:43,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 19:22:43,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 19:22:43,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 19:22:43,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 19:22:43,585 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-01 19:22:43,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 19:22:43,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 19:22:43,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 19:22:43,638 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-01 19:22:43,638 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 ############################################################ # # Step 3: Create 40000000 items sequentially # ############################################################ ############################################################ # # RAM FootPrint: 7300.03125 # ############################################################ Read Start: 0 Read End: 0 Update Start: 0 Update End: 0 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 40000000 Create End: 80000000 Final Start: 40000000 Final End: 80000000 2021-12-01 19:22:43,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:43,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1638415364.7 2021-12-01 19:22:44,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:44,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1638415365.71 2021-12-01 19:22:45,721 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:45,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1638415366.72 2021-12-01 19:22:46,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:46,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1638415367.73 2021-12-01 19:22:47,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:47,744 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1638415368.74 2021-12-01 19:22:48,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:48,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1638415369.76 2021-12-01 19:22:49,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:49,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1638415370.79 2021-12-01 19:22:50,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:50,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1638415371.82 2021-12-01 19:22:51,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:51,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1638415372.83 2021-12-01 19:22:52,839 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:52,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1638415373.84 2021-12-01 19:22:53,852 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:53,852 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:22:54,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:54,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:22:55,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:55,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:22:56,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:56,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:22:57,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:57,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:22:58,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:58,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:22:59,959 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:22:59,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:00,971 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:00,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:01,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:01,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:02,994 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:02,994 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:04,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:04,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:05,020 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:05,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:06,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:06,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:07,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:07,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:08,114 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:08,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:09,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:09,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:10,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:10,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:11,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:11,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:12,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:12,178 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:13,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:13,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:14,206 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:14,206 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:15,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:15,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:16,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:16,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:17,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:17,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:18,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:18,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:19,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:19,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:20,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:20,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:21,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:21,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:22,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:22,318 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:23,329 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:23,329 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:24,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:24,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:25,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:25,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:26,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:26,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:27,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:27,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:28,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:28,392 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:29,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:29,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:30,418 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:30,420 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:31,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:31,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:32,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:32,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 19:23:33,459 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 19:23:33,459 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection40_2_1638415372.83 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection35_0_1638415374.85 Loader_GleamBookUsers0__default_VolumeCollection46_10_1638415364.7 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638415375.87 Loader_GleamBookUsers0__default_VolumeCollection48_6_1638415368.74 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638415376.88 Loader_GleamBookUsers0__default_VolumeCollection42_4_1638415370.79 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638415377.89 Loader_GleamBookUsers0__default_VolumeCollection44_8_1638415366.72 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638415378.9 Loader_GleamBookUsers0__default_VolumeCollection47_9_1638415365.71 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638415379.95 Loader_GleamBookUsers0__default_VolumeCollection41_1_1638415373.84 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638415380.96 Loader_GleamBookUsers0__default_VolumeCollection49_5_1638415369.76 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638415381.97 Loader_GleamBookUsers0__default_VolumeCollection45_7_1638415367.73 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638415382.98 Loader_GleamBookUsers0__default_VolumeCollection43_3_1638415371.82 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638415384.0 Loader_GleamBookUsers0__default_VolumeCollection35_0_1638415374.85 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638415385.01 Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638415375.87 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638415386.07 Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638415376.88 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638415387.08 Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638415377.89 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638415388.1 Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638415379.95 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638415389.12 Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638415384.0 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638415390.13 Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638415381.97 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638415391.15 Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638415378.9 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638415392.16 Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638415380.96 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638415393.18 Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638415386.07 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638415394.19 Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638415382.98 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638415395.21 Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638415388.1 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638415396.22 Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638415385.01 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638415397.23 Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638415390.13 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638415398.25 Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638415387.08 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638415399.26 Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638415392.16 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638415400.28 Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638415389.12 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638415401.29 Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638415394.19 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638415402.31 Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638415391.15 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638415403.32 Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638415398.25 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638415404.33 Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638415395.21 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638415405.35 Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638415393.18 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638415406.36 Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638415396.22 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638415407.37 Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638415400.28 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638415408.38 Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638415402.31 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638415409.4 Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638415397.23 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638415410.41 Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638415399.26 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638415411.42 Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638415401.29 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638415412.43 Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638415404.33 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638415413.45 Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638415406.36 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638415414.56 Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638415403.32 is completed! Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638415410.41 is completed! Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638415405.35 is completed! Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638415407.37 is completed! Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638415408.38 is completed! Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638415409.4 is completed! Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638415411.42 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638415414.56 is completed! Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638415412.43 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638415413.45 is completed! 2021-12-01 23:28:02,736 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-01 23:28:02,740 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 23:28:02,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:03,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 23:28:03,209 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:03,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 23:28:03,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 23:28:03,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 23:28:03,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 23:28:03,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 23:28:03,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 23:28:03,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 23:28:03,355 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 23:28:03,355 | infra | INFO | pool-3-thread-14 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.36' started 2021-12-01 23:28:03,355 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.36 2021-12-01 23:28:03,358 | infra | DEBUG | pool-3-thread-14 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-01 23:28:03,361 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 23:28:03,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:03,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 23:28:03,801 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:03,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 23:28:03,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 23:28:03,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 23:28:03,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 23:28:03,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 23:28:03,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 23:28:03,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 23:28:03,903 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 23:28:03,903 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.9 2021-12-01 23:28:03,904 | infra | INFO | pool-3-thread-15 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.9' started 2021-12-01 23:28:03,907 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-01 23:28:03,910 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 23:28:04,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:04,092 | test | DEBUG | pool-3-thread-14 | [task:_compare:2725] Comparing 0 == 0 2021-12-01 23:28:04,092 | test | DEBUG | pool-3-thread-14 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.64': 0} for bucket 'GleamBookUsers0' 2021-12-01 23:28:04,092 | infra | INFO | pool-3-thread-14 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.36' completed 2021-12-01 23:28:04,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 23:28:04,408 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:04,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 23:28:04,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 23:28:04,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 23:28:04,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 23:28:04,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 23:28:04,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 23:28:04,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 23:28:04,582 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-01 23:28:04,582 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430084.58 2021-12-01 23:28:04,584 | infra | INFO | pool-3-thread-16 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430084.58' started 2021-12-01 23:28:04,585 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.36 2021-12-01 23:28:04,585 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 23:28:04,585 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-01 23:28:04,585 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.9 2021-12-01 23:28:04,690 | test | DEBUG | pool-3-thread-15 | [task:_compare:2725] Comparing 0 == 0 2021-12-01 23:28:04,690 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.65': 0} for bucket 'GleamBookUsers0' 2021-12-01 23:28:04,690 | infra | INFO | pool-3-thread-15 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430083.9' completed 2021-12-01 23:28:04,691 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 23:28:04,691 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430084.58 2021-12-01 23:28:05,372 | test | DEBUG | pool-3-thread-16 | [task:_compare:2725] Comparing 0 == 0 2021-12-01 23:28:05,372 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.66': 0} for bucket 'GleamBookUsers0' 2021-12-01 23:28:05,374 | infra | INFO | pool-3-thread-16 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638430084.58' completed 2021-12-01 23:28:05,375 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 23:28:05,415 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3946] Stats: {u'opsPerSec': 19340.7, u'diskFetches': 0, u'dataUsed': 8173590473363L, u'itemCount': 3999973938L, u'quotaPercentUsed': 79.26818212121725, u'storageTotals': {u'hdd': {u'usedByData': 11318864466026L, u'used': 11432856681180L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 24542953309476L}, u'ram': {u'usedByData': 255340687368L, u'used': 375047712768L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 11318864466026L, u'vbActiveNumNonResident': 3866172293L, u'memUsed': 255340687368L} 2021-12-01 23:28:05,766 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 3999973938 | 300.00 GiB | 237.80 GiB | 10.29 TiB | 3.34506291976 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ 2021-12-01 23:28:05,766 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+-------------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+-------------------+---------------+---------------+ | 2000000000 | 4000000000 | 0-0 | 40000000-80000000 | 0-0 | 0-0 | +---------------+---------------+---------------+-------------------+---------------+---------------+ 2021-12-01 23:28:06,128 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 3.34506291976 2021-12-01 23:28:06,131 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 23:28:06,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:06,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 23:28:06,592 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:06,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 23:28:06,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 23:28:06,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 23:28:06,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 23:28:06,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 23:28:06,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 23:28:06,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 23:28:06,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 23:28:07,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 23:28:07,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 23:28:07,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 23:28:07,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 23:28:09,232 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 23:28:09,236 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 23:28:09,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:09,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 23:28:09,703 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:09,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 23:28:09,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 23:28:09,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 23:28:09,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 23:28:09,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 23:28:09,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 23:28:09,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 23:28:09,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 23:28:10,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 23:28:10,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 23:28:10,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 23:28:10,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 23:28:12,357 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 23:28:12,361 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 23:28:12,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:12,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 23:28:12,852 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:12,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 23:28:12,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 23:28:12,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 23:28:12,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 23:28:12,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 23:28:12,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 23:28:12,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 23:28:13,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-01 23:28:13,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-01 23:28:13,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-01 23:28:13,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-01 23:28:13,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-01 23:28:15,598 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 23:28:15,598 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 10834726MB 2021-12-01 23:28:15,599 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 7824MB 2021-12-01 23:28:15,601 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 351884MB 2021-12-01 23:28:15,602 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 10482432MB 2021-12-01 23:28:15,605 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 23:28:15,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:15,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 23:28:16,076 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:16,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 23:28:16,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 23:28:16,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 23:28:16,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 23:28:16,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 23:28:16,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 23:28:16,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 23:28:16,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 23:28:16,282 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-01 23:28:16,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 23:28:16,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 23:28:17,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 23:28:17,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 23:28:17,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 23:28:18,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 23:28:18,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 23:28:18,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 23:28:19,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 23:28:19,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 23:28:20,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 23:28:20,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 23:28:21,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 23:28:21,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 23:28:22,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 23:28:22,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 23:28:23,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 23:28:23,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 23:28:23,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 23:28:24,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 23:28:24,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 23:28:25,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 23:28:25,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 23:28:26,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 23:28:26,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 23:28:27,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 23:28:27,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 23:28:28,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 23:28:28,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 23:28:29,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 23:28:29,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 23:28:29,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 23:28:30,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 23:28:30,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 23:28:31,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 23:28:31,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 23:28:32,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 23:28:32,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 23:28:33,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 23:28:33,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 23:28:34,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 23:28:34,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 23:28:34,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 23:28:35,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 23:28:35,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 23:28:36,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 23:28:36,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 23:28:37,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 23:28:37,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 23:28:37,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 23:28:38,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 23:28:38,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 23:28:39,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 23:28:39,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 23:28:39,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 23:28:40,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 23:28:40,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 23:28:41,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 23:28:41,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 23:28:42,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 23:28:42,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 23:28:42,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 23:28:43,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 23:28:43,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 23:28:43,755 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 23:28:43,757 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 23:28:43,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:28:44,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 23:28:44,236 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:28:44,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 23:28:44,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 23:28:44,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 23:28:44,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 23:28:44,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 23:28:44,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 23:28:44,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 23:28:44,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 23:28:44,441 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-01 23:28:44,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 23:28:44,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 23:28:45,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 23:28:45,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 23:28:46,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 23:28:46,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 23:28:46,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 23:28:47,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 23:28:47,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 23:28:48,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 23:28:48,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 23:28:48,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 23:28:49,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 23:28:49,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 23:28:50,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 23:28:50,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 23:28:50,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 23:28:51,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 23:28:51,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 23:28:52,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 23:28:52,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 23:28:53,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 23:28:53,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 23:28:53,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 23:28:54,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 23:28:54,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 23:28:55,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 23:28:55,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 23:28:56,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 23:28:56,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 23:28:56,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 23:28:57,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 23:28:57,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 23:28:58,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 23:28:58,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 23:28:59,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 23:28:59,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 23:28:59,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 23:29:00,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 23:29:00,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 23:29:01,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 23:29:01,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 23:29:02,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 23:29:02,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 23:29:03,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 23:29:03,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 23:29:03,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 23:29:04,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 23:29:04,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 23:29:05,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 23:29:05,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 23:29:06,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 23:29:06,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 23:29:07,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 23:29:07,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 23:29:08,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 23:29:08,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 23:29:09,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 23:29:09,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 23:29:10,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 23:29:10,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 23:29:11,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 23:29:11,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 23:29:11,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 23:29:12,289 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-01 23:29:12,293 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 23:29:12,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:29:12,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 23:29:12,786 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:29:12,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 23:29:12,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 23:29:12,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 23:29:12,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 23:29:12,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 23:29:12,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 23:29:12,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 23:29:12,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-01 23:29:13,036 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-01 23:29:13,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-01 23:29:13,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-01 23:29:13,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-01 23:29:14,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-01 23:29:14,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-01 23:29:15,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-01 23:29:15,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-01 23:29:16,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-01 23:29:16,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-01 23:29:16,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-01 23:29:17,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-01 23:29:17,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-01 23:29:18,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-01 23:29:18,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-01 23:29:19,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-01 23:29:19,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-01 23:29:20,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-01 23:29:20,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-01 23:29:21,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-01 23:29:21,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-01 23:29:22,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-01 23:29:22,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-01 23:29:22,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-01 23:29:23,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-01 23:29:23,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-01 23:29:24,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-01 23:29:24,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-01 23:29:24,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-01 23:29:25,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-01 23:29:25,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-01 23:29:26,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-01 23:29:26,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-01 23:29:27,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-01 23:29:27,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-01 23:29:28,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-01 23:29:28,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-01 23:29:29,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-01 23:29:29,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-01 23:29:30,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-01 23:29:30,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-01 23:29:30,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-01 23:29:31,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-01 23:29:31,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-01 23:29:32,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-01 23:29:32,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-01 23:29:33,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-01 23:29:33,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-01 23:29:34,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-01 23:29:34,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-01 23:29:35,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-01 23:29:35,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-01 23:29:35,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-01 23:29:36,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-01 23:29:36,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-01 23:29:37,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-01 23:29:37,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-01 23:29:38,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-01 23:29:38,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-01 23:29:39,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-01 23:29:39,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-01 23:29:40,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-01 23:29:40,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-01 23:29:41,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-01 23:29:41,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-01 23:29:42,012 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 23:29:42,013 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 27898.5, u'ReplayWALOffset': u'7147:9367552', u'NTablesCreated': 607405, u'RecentReadIOAmpSet': 0.022602416574954987, u'ReadIOAmp': 260.45208740234375, u'BlockCacheMisses': 32195354, u'NFlushes': 152017, u'RecentWriteAmp': 4.346617698669434, u'NumSyncFlushes': 1190, u'TotalMemUsed': 128207800, u'NBloomFilterHits': 789112675, u'Fragmentation': 0.26717230677604675, u'NumNormalFlushes': 78947, u'RecentBlockCacheHitRatio': 62.07989501953125, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 286926, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 6153240, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 403954899, u'CheckpointOverhead': 0, u'NItemsGet': 0, u'NWriteIOs': 449880, u'NReadBytes': 17770993519L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 22390577, u'SizeRatio': 0.4853026866912842, u'NWriteBytes': 730820608, u'BloomFilterMemUsed': 0, u'NumTables': 55, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 749568}, {u'TargetSize': 46137344, u'LogicalSize': 28456156, u'SizeRatio': 0.6167705709284003, u'NWriteBytes': 580325376, u'BloomFilterMemUsed': 0, u'NumTables': 19, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 372736}, {u'TargetSize': 570541359, u'LogicalSize': 282076328, u'SizeRatio': 0.4944011920439934, u'NWriteBytes': 1717956608, u'BloomFilterMemUsed': 0, u'NumTables': 144, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 3395584}, {u'TargetSize': 29184532698L, u'LogicalSize': 15883439486L, u'SizeRatio': 0.5442416930351769, u'NWriteBytes': 3577847808L, u'BloomFilterMemUsed': 0, u'NumTables': 423, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 150401024}, {u'TargetSize': 114108273398L, u'LogicalSize': 58369065397L, u'SizeRatio': 0.5115235176104509, u'NWriteBytes': 70896177152L, u'BloomFilterMemUsed': 0, u'NumTables': 242, u'WastedSpace': 1314928698, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 58369064960L}], u'NItemsGarbageCollected': 11061212, u'NInternalFragmentationCompacts': 33, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 351333270, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 89760, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 6041725, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 263670, u'CompactWriter': {u'NDataBlockWrites': 2089270, u'NWriteBytes': 6157361152L, u'NIndexBlockWrites': 111537, u'NBytesIncoming': 8511254363L, u'NSyncs': 0, u'NInserts': 360366763, u'NWriteIOs': 105309}, u'NWriteBytes': 77536309472L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 11061212, u'LogicalDiskSize': 58369065397L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.8314528017855712, u'BufferMemUsed': 479590, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 42485625911L, u'NTableFiles': 883, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 231880, u'TreeSnapshotMemoryUsed': 132208, u'TotalDiskUsage': 67372199936L, u'BytesIncoming': 93254011900L, u'NItemsDel': 0, u'PendingCheckpointBytes': 33366016, u'NCompacts': 31523, u'NItemsPut': 85736900, u'NSyncs': 73070, u'CompactReader': {u'NReadIOs': 6041725, u'NReadBytes': 17770993519L, u'NDataBlockReads': 5866300, u'NBytesOutgoing': 0, u'NIndexBlockReads': 175425}, u'ActiveDiskUsage': 58523983872L, u'NTablesDeleted': 142454, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 21621202, u'NWriteBytes': 71378948320L, u'NIndexBlockWrites': 575766, u'NBytesIncoming': 94019732811L, u'NSyncs': 73070, u'NInserts': 85732837, u'NWriteIOs': 344571}}, u'NReadBytes': 13679900840289L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 303.1333312988281, u'NInternalFragmentationCompacts': 33, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 375320295, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 5068, u'NTxns': 6814901, u'NWriteBytes': 113121427456L, u'NSegments': 7149, u'NSyncs': 6814901, u'NLoggedBytes': 113121427456L, u'NActiveSegments': 3, u'NWrites': 6829309, u'NInserts': 125310351, u'DiskUsed': 42422272}, u'NGets': 19497058, u'NReadBytesSet': 33395671743L, u'NReadBytesCompact': 65993295453L, u'IndexResidentRatio': 0.04219229519367218, u'CompactQueueSize': 0, u'NSyncsPerSec': 0.8999999761581421, u'BlockCacheHitRatio': 74.92744445800781, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 13613907544836L, u'BloomFilterMemUsedRatio': 0.26079338788986206, u'NWriteBytes': 244213584772L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.114255391061306, u'WriteAmp': 2.5368237495422363, u'NBlocksCached': 27239, u'LogicalDataSize': 43933039214L, u'ActiveBloomFilterMemUsed': 38306200, u'NFileCountCompacts': 17007, u'NTableFiles': 1925, u'FlushQueueSize': 72, u'TableObjectMemUsed': 490296, u'BlockCacheQuota': 87296067, u'TotalDiskUsage': 69185826816L, u'NPurgerRuns': 132, u'ReadIOAmpSet': 0.0, u'NSyncs': 219210, u'NReadIOsSetPerSec': 4.800000190734863, u'NReadIOsPerSec': 304.3666687011719, u'NTablesDeleted': 462290, u'BloomFilterFPR': 0.01651875674724579, u'NBloomFilterHitsPerSec': 2802.300048828125, u'WriteCacheMemUsed': 8694856, u'RecentBloomFilterFPR': 0.02165716327726841, u'BytesPerRead': 696541.5625, u'BytesIncomingPerSec': 398697.6875, u'NReadIOsGetPerSec': 10.766666412353516, u'NGetStats': 405975, u'TotalIndexBlocksSize': 428770210, u'NGetStatsPerSec': 4.566666603088379, u'NWriteBytesCompact': 53178306560L, u'NReadIOsSet': 12563943, u'ActiveDataSize': 43933039214L, u'NWriteBytesCompactPerSec': 769228.8125, u'NBloomFilterMissesPerSec': 62.03333282470703, u'NCommitBatches': 6814901, u'NGetStatsComputed': 405975, u'NWriteIOs': 7827508, u'NReadIOsGet': 5090613189L, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 21824018, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 212.36666870117188, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.6666666865348816, u'ManagedFileObjectMemUsed': 189792, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 78108173, u'NWriterCompacts': 0, u'NReadIOs': 5114907704L, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 1126593024, u'NTablesCreated': 167704, u'ReadIOAmp': 60.74105183173218, u'MemtablesMemUsed': 369688, u'NBloomFilterHits': 789112675, u'TotalIndexBlocksSize': 24799891, u'CheckpointOverhead': 32198656, u'NItemsGet': 83808158, u'NWriteIOs': 322474, u'NReadBytes': 13661952832536L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 2510752, u'SizeRatio': 0.054419084028764206, u'NWriteBytes': 5206331392L, u'BloomFilterMemUsed': 33416, u'NumTables': 100, u'WastedSpace': 28, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2392064}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 27065237, u'SizeRatio': 0.5866232135079124, u'NWriteBytes': 21150195712L, u'BloomFilterMemUsed': 831744, u'NumTables': 73, u'WastedSpace': 42240, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 27004928}, {u'TargetSize': 158083289, u'LogicalSize': 133545146, u'SizeRatio': 0.8447771225205214, u'NWriteBytes': 10362413056L, u'BloomFilterMemUsed': 4079808, u'NumTables': 93, u'WastedSpace': 331323, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 133832704}, {u'TargetSize': 1580832927, u'LogicalSize': 1580832927, u'SizeRatio': 1.0, u'NWriteBytes': 15838908416L, u'BloomFilterMemUsed': 33359880, u'NumTables': 285, u'WastedSpace': 5377605, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1586208768}], u'NItemsGarbageCollected': 21425800, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 23982477, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 69792, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 5108543093L, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 220953, u'CompactWriter': {u'NDataBlockWrites': 17154940, u'NWriteBytes': 46951452672L, u'NIndexBlockWrites': 223041, u'NBytesIncoming': 65342312392L, u'NSyncs': 0, u'NInserts': 1126593024, u'NWriteIOs': 105011}, u'NWriteBytes': 52266792861L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 5090595669L, u'NReadBytes': 13613882079037L, u'NDataBlockReads': 11541843537L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 22348724689L}, u'NBloomFilterMisses': 13254101, u'NItemsMerged': 0, u'LogicalDiskSize': 1580832927, u'TotalBloomFilterMemUsed': 38890576, u'WriteAmp': 16.586097468611694, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 38304848, u'LogicalDataSize': 1447287781, u'NTableFiles': 727, u'NFileCountCompacts': 8524, u'TableObjectMemUsed': 180296, u'TreeSnapshotMemoryUsed': 102832, u'TotalDiskUsage': 1811046400, u'BytesIncoming': 3151241150L, u'NItemsDel': 0, u'PendingCheckpointBytes': 58761216, u'NCompacts': 10783, u'NItemsPut': 64311100, u'NSyncs': 73070, u'CompactReader': {u'NReadIOs': 17947424, u'NReadBytes': 48070753499L, u'NDataBlockReads': 17587689, u'NBytesOutgoing': 0, u'NIndexBlockReads': 359735}, u'ActiveDiskUsage': 1749438464, u'NTablesDeleted': 167153, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.016518756005919167, u'Writer': {u'NDataBlockWrites': 1082982, u'NWriteBytes': 5315340189L, u'NIndexBlockWrites': 144627, u'NBytesIncoming': 3729805396L, u'NSyncs': 73070, u'NInserts': 64307037, u'NWriteIOs': 217463}}, u'NReadBytesCompactPerSec': 755318.6875, u'TableMetaMemUsed': 575013, u'NDeletes': 0, u'BlockCacheHitsPerSec': 496.26666259765625, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 13254101, u'LogicalDiskSize': 59950023846L, u'NSets': 64311100, u'TotalBloomFilterMemUsed': 38892328, u'BufferMemUsed': 479590, u'RecentReadAmp': 0.0, u'NGetsPerSec': 94.23332977294922, u'NInserts': 0, u'NCommitBatchesPerSec': 176.73333740234375, u'FragmentationWithOverhead': 0.3649994134902954, u'NCompactsPerSec': 0.23333333432674408, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 783217.125, u'TreeSnapshotMemoryUsed': 280560, u'BytesIncoming': 96267460842L, u'NCompacts': 50789, u'NumPersistentFlushes': 71880, u'NWriteIOsPerSec': 181.53334045410156, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 296.0576477050781, u'ActiveDiskUsage': 60274176000L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 4.566666603088379, u'NReadBytesSetPerSec': 12506.06640625, u'BlockCacheHits': 96213385, u'NWriteBytesPerSec': 1732986.375, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 152775, u'ReadIOAmp': 0.00128333879045319, u'MemtablesMemUsed': 2171928, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 15420, u'CheckpointOverhead': 1531904, u'NItemsGet': 13651890, u'NWriteIOs': 225845, u'NReadBytes': 177014234, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 733905, u'SizeRatio': 0.01590696248141202, u'NWriteBytes': 1182085120, u'BloomFilterMemUsed': 648, u'NumTables': 81, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 663552}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 125522, u'SizeRatio': 0.002720616080544212, u'NWriteBytes': 69492736, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 40547075, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 4548, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 30240, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 322886, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 90390, u'CompactWriter': {u'NDataBlockWrites': 8483, u'NWriteBytes': 69492736, u'NIndexBlockWrites': 8483, u'NBytesIncoming': 38505077, u'NSyncs': 0, u'NInserts': 449599, u'NWriteIOs': 8483}, u'NWriteBytes': 1289054983, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 17520, u'NReadBytes': 25465799, u'NDataBlockReads': 6803285, u'NBytesOutgoing': 0, u'NIndexBlockReads': 6803285}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 125522, u'TotalBloomFilterMemUsed': 1752, u'WriteAmp': 0.31721077306115586, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1352, u'LogicalDataSize': 125522, u'NTableFiles': 315, u'NFileCountCompacts': 8483, u'TableObjectMemUsed': 78120, u'TreeSnapshotMemoryUsed': 45520, u'TotalDiskUsage': 2580480, u'BytesIncoming': 4063717542L, u'NItemsDel': 0, u'PendingCheckpointBytes': 327680, u'NCompacts': 8483, u'NItemsPut': 40554562, u'NSyncs': 73070, u'CompactReader': {u'NReadIOs': 305366, u'NReadBytes': 151548435, u'NDataBlockReads': 152683, u'NBytesOutgoing': 0, u'NIndexBlockReads': 152683}, u'ActiveDiskUsage': 753664, u'NTablesDeleted': 152683, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 144292, u'NWriteBytes': 1219562247, u'NIndexBlockWrites': 144292, u'NBytesIncoming': 107664417, u'NSyncs': 73070, u'NInserts': 1401980, u'NWriteIOs': 217362}}}}} 2021-12-01 23:29:42,017 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.26717230677604675, 0.26966872811317444, 0.27099689841270447, 0.2712387144565582, 0.2698584198951721, 0.27338850498199463, 0.272088885307312, 0.2725244164466858, 0.27154380083084106, 0.27637338638305664, 0.273430198431015, 0.27471795678138733, 0.2719164490699768, 0.2722247540950775, 0.2720230221748352, 0.2713126838207245, 0.2712197005748749, 0.2732507586479187, 0.2718661427497864, 0.2734212279319763, 0.2755495309829712, 0.2702828645706177, 0.27385225892066956, 0.2758881449699402, 0.2766515016555786, 0.2730448544025421, 0.27125683426856995, 0.27276208996772766, 0.2681541442871094, 0.2728855311870575, 0.2712186276912689, 0.2733805775642395, 0.27213162183761597, 0.27228212356567383, 0.2739677429199219, 0.2715042531490326, 0.26718762516975403, 0.2688038647174835, 0.26919469237327576, 0.2675807774066925, 0.27470335364341736, 0.26790717244148254, 0.2741203308105469, 0.27179351449012756, 0.27285391092300415, 0.2700207233428955, 0.26604071259498596, 0.26855573058128357, 0.27383559942245483, 0.26673799753189087, 0.2671404182910919, 0.27322837710380554, 0.2721586525440216, 0.26348963379859924, 0.2719159424304962, 0.272703617811203, 0.2744605243206024, 0.2713673710823059, 0.27147090435028076, 0.2731764316558838, 0.26921701431274414, 0.2740108072757721, 0.26772400736808777, 0.2717118561267853], '172.23.110.66': [0.27430376410484314, 0.2724369764328003, 0.27189722657203674, 0.2688954770565033, 0.273001104593277, 0.2767811417579651, 0.2713577151298523, 0.2697407901287079, 0.27154281735420227, 0.27364715933799744, 0.2735995948314667, 0.27220192551612854, 0.27659323811531067, 0.26803308725357056, 0.27508237957954407, 0.27201950550079346, 0.2717548608779907, 0.272646427154541, 0.2746811509132385, 0.2770567536354065, 0.2720875144004822, 0.2750941216945648, 0.27348628640174866, 0.27513065934181213, 0.27487021684646606, 0.27031636238098145, 0.27240005135536194, 0.27471283078193665, 0.27051597833633423, 0.27481314539909363, 0.27609238028526306, 0.2749500572681427, 0.272358775138855, 0.27280277013778687, 0.27497169375419617, 0.26955756545066833, 0.2737216353416443, 0.273054838180542, 0.27329757809638977, 0.27127277851104736, 0.27284979820251465, 0.2723167836666107, 0.28024041652679443, 0.2694976031780243, 0.2715397775173187, 0.27267101407051086, 0.2723013460636139, 0.2698461413383484, 0.2727219760417938, 0.2717038094997406, 0.2736717164516449, 0.27237388491630554, 0.27292415499687195, 0.2689853310585022, 0.2787993848323822, 0.27201178669929504, 0.27004969120025635, 0.27535340189933777, 0.275543212890625, 0.26887568831443787, 0.27122870087623596, 0.2718653976917267, 0.2701369822025299, 0.2754764258861542], '172.23.110.64': [0.2751578986644745, 0.2725180983543396, 0.2725996673107147, 0.272032231092453, 0.27160584926605225, 0.2719644606113434, 0.2719409763813019, 0.26884725689888, 0.26918825507164, 0.2740488350391388, 0.26969820261001587, 0.2704392373561859, 0.2699264883995056, 0.2720470130443573, 0.27330055832862854, 0.27126625180244446, 0.27332088351249695, 0.26947546005249023, 0.2683144211769104, 0.27454429864883423, 0.2683924436569214, 0.26846009492874146, 0.2658545672893524, 0.27432525157928467, 0.2725927233695984, 0.2727634012699127, 0.26587268710136414, 0.26990944147109985, 0.27120527625083923, 0.2704262137413025, 0.26946496963500977, 0.2711156904697418, 0.2682006359100342, 0.27195075154304504, 0.27265694737434387, 0.270295649766922, 0.2722350060939789, 0.26966607570648193, 0.27424487471580505, 0.27509450912475586, 0.27275171875953674, 0.2696770429611206, 0.27432936429977417, 0.2724132239818573, 0.27346503734588623, 0.2758690118789673, 0.27701064944267273, 0.27279841899871826, 0.2717286944389343, 0.27150872349739075, 0.27118727564811707, 0.2714857757091522, 0.27258923649787903, 0.2721998691558838, 0.27041882276535034, 0.2701268196105957, 0.2711337208747864, 0.2728468179702759, 0.2671411633491516, 0.27339115738868713, 0.2734895646572113, 0.2681798040866852, 0.274287611246109, 0.2704416811466217]} 2021-12-01 23:29:42,367 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 28.0 2021-12-01 23:29:42,701 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 28.0 2021-12-01 23:29:43,059 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 28.0 2021-12-01 23:29:43,059 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 28.0, '172.23.110.66': 28.0, '172.23.110.64': 28.0} 2021-12-01 23:29:43,062 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-01 23:29:43,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:29:43,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-01 23:29:43,546 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:29:43,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-01 23:29:43,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-01 23:29:43,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-01 23:29:43,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-01 23:29:43,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-01 23:29:43,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-01 23:29:43,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-01 23:29:43,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 23:29:43,749 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-01 23:29:43,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 23:29:43,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:43,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 23:29:43,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 23:29:43,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 23:29:43,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:44,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 23:29:44,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 23:29:44,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 23:29:44,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 23:29:44,157 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-01 23:29:44,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 23:29:44,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:44,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 23:29:44,226 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-01 23:29:44,226 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-01 23:29:44,230 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-01 23:29:44,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:29:44,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-01 23:29:44,706 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:29:44,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-01 23:29:44,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-01 23:29:44,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-01 23:29:44,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-01 23:29:44,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-01 23:29:44,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-01 23:29:44,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-01 23:29:44,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 23:29:44,858 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-01 23:29:44,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 23:29:44,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:44,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 23:29:45,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 23:29:45,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 23:29:45,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:45,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 23:29:45,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 23:29:45,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 23:29:45,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 23:29:45,198 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-01 23:29:45,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 23:29:45,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:45,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 23:29:45,247 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-01 23:29:45,249 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-01 23:29:45,250 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-01 23:29:45,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-01 23:29:45,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-01 23:29:45,737 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-01 23:29:45,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-01 23:29:45,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-01 23:29:45,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-01 23:29:45,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-01 23:29:45,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-01 23:29:45,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-01 23:29:45,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-01 23:29:45,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-01 23:29:45,897 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-01 23:29:45,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-01 23:29:45,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:46,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 23:29:46,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 23:29:46,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-01 23:29:46,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:46,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-01 23:29:46,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 23:29:46,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-01 23:29:46,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-01 23:29:46,197 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-01 23:29:46,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-01 23:29:46,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-01 23:29:46,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-01 23:29:46,242 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-01 23:29:46,242 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 ############################################################ # # Step 3.1: Update 40000000 RandonKey keys to create 50 percent fragmentation # ############################################################ ############################################################ # # RAM FootPrint: 6688.0234375 # ############################################################ Read Start: 0 Read End: 0 Update Start: 40000000 Update End: 80000000 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 0 Create End: 0 Final Start: 40000000 Final End: 80000000 2021-12-01 23:29:46,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:46,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1638430187.29 2021-12-01 23:29:47,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:47,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1638430188.3 2021-12-01 23:29:48,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:48,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1638430189.31 2021-12-01 23:29:49,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:49,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1638430190.32 2021-12-01 23:29:50,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:50,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1638430191.33 2021-12-01 23:29:51,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:51,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1638430192.34 2021-12-01 23:29:52,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:52,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1638430193.4 2021-12-01 23:29:53,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:53,413 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1638430194.41 2021-12-01 23:29:54,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:54,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1638430195.43 2021-12-01 23:29:55,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:55,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1638430196.49 2021-12-01 23:29:56,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:56,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:29:57,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:57,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:29:58,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:58,553 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:29:59,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:29:59,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:00,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:00,579 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:01,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:01,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:02,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:02,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:03,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:03,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:04,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:04,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:05,660 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:05,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:06,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:06,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:07,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:07,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:08,698 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:08,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:09,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:09,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:10,721 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:10,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:11,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:11,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:12,747 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:12,749 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:13,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:13,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:14,775 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:14,775 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:15,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:15,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:16,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:16,798 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:17,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:17,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:18,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:18,826 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:19,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:19,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:20,852 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:20,852 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:21,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:21,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:22,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:22,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:23,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:23,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:24,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:24,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:25,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:25,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:26,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:26,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:27,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:27,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:28,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:28,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:30,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:30,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:31,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:31,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:32,076 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:32,076 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:33,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:33,091 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:34,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:34,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:35,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:35,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-01 23:30:36,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-01 23:30:36,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection40_2_1638430195.43 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection35_0_1638430197.53 Loader_GleamBookUsers0__default_VolumeCollection43_3_1638430194.41 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638430198.54 Loader_GleamBookUsers0__default_VolumeCollection46_10_1638430187.29 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638430199.55 Loader_GleamBookUsers0__default_VolumeCollection48_6_1638430191.33 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638430200.57 Loader_GleamBookUsers0__default_VolumeCollection44_8_1638430189.31 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638430201.58 Loader_GleamBookUsers0__default_VolumeCollection42_4_1638430193.4 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638430202.59 Loader_GleamBookUsers0__default_VolumeCollection47_9_1638430188.3 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638430203.6 Loader_GleamBookUsers0__default_VolumeCollection49_5_1638430192.34 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638430204.64 Loader_GleamBookUsers0__default_VolumeCollection45_7_1638430190.32 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638430205.65 Loader_GleamBookUsers0__default_VolumeCollection41_1_1638430196.49 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638430206.66 Loader_GleamBookUsers0__default_VolumeCollection35_0_1638430197.53 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638430207.67 Loader_GleamBookUsers0__default_VolumeCollection33_-2_1638430199.55 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638430208.69 Loader_GleamBookUsers0__default_VolumeCollection36_-1_1638430198.54 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638430209.7 Loader_GleamBookUsers0__default_VolumeCollection34_-3_1638430200.57 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638430210.71 Loader_GleamBookUsers0__default_VolumeCollection39_-4_1638430201.58 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638430211.72 Loader_GleamBookUsers0__default_VolumeCollection37_-5_1638430202.59 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638430212.74 Loader_GleamBookUsers0__default_VolumeCollection38_-6_1638430203.6 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638430213.75 Loader_GleamBookUsers0__default_VolumeCollection7_-7_1638430204.64 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638430214.76 Loader_GleamBookUsers0__default_VolumeCollection6_-8_1638430205.65 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638430215.78 Loader_GleamBookUsers0__default_VolumeCollection5_-9_1638430206.66 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638430216.79 Loader_GleamBookUsers0__default_VolumeCollection4_-10_1638430207.67 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638430217.8 Loader_GleamBookUsers0__default_VolumeCollection3_-11_1638430208.69 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638430218.81 Loader_GleamBookUsers0__default_VolumeCollection31_-12_1638430209.7 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638430219.83 Loader_GleamBookUsers0__default_VolumeCollection2_-13_1638430210.71 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638430220.84 Loader_GleamBookUsers0__default_VolumeCollection32_-14_1638430211.72 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638430221.85 Loader_GleamBookUsers0__default_VolumeCollection1_-15_1638430212.74 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638430222.87 Loader_GleamBookUsers0__default_VolumeCollection0_-16_1638430213.75 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638430223.92 Loader_GleamBookUsers0__default_VolumeCollection30_-17_1638430214.76 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638430224.93 Loader_GleamBookUsers0__default_VolumeCollection9_-18_1638430215.78 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638430225.95 Loader_GleamBookUsers0__default_VolumeCollection8_-19_1638430216.79 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638430226.96 Loader_GleamBookUsers0__default_VolumeCollection19_-20_1638430217.8 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638430227.97 Loader_GleamBookUsers0__default_VolumeCollection25_-22_1638430219.83 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638430228.98 Loader_GleamBookUsers0__default_VolumeCollection24_-21_1638430218.81 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638430230.0 Loader_GleamBookUsers0__default_VolumeCollection22_-23_1638430220.84 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638430231.06 Loader_GleamBookUsers0__default_VolumeCollection23_-24_1638430221.85 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638430232.07 Loader_GleamBookUsers0__default_VolumeCollection29_-26_1638430223.92 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638430233.08 Loader_GleamBookUsers0__default_VolumeCollection28_-25_1638430222.87 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638430234.09 Loader_GleamBookUsers0__default_VolumeCollection26_-27_1638430224.93 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638430235.11 Loader_GleamBookUsers0__default_VolumeCollection27_-28_1638430225.95 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638430236.12 Loader_GleamBookUsers0__default_VolumeCollection20_-29_1638430226.96 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638430237.13 Loader_GleamBookUsers0__default_VolumeCollection21_-30_1638430227.97 is completed! Loader_GleamBookUsers0__default_VolumeCollection14_-32_1638430230.0 is completed! Loader_GleamBookUsers0__default_VolumeCollection12_-34_1638430232.07 is completed! Loader_GleamBookUsers0__default_VolumeCollection13_-31_1638430228.98 is completed! Loader_GleamBookUsers0__default_VolumeCollection11_-33_1638430231.06 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_-38_1638430236.12 is completed! Loader_GleamBookUsers0__default_VolumeCollection17_-35_1638430233.08 is completed! Loader_GleamBookUsers0__default_VolumeCollection15_-37_1638430235.11 is completed! Loader_GleamBookUsers0__default_VolumeCollection18_-36_1638430234.09 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-39_1638430237.13 is completed! 2021-12-02 03:13:46,660 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-02 03:13:46,664 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 03:13:46,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:13:47,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 03:13:47,119 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:13:47,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 03:13:47,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 03:13:47,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 03:13:47,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 03:13:47,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 03:13:47,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 03:13:47,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 03:13:47,243 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-02 03:13:47,244 | infra | INFO | pool-3-thread-17 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.24' started 2021-12-02 03:13:47,244 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.24 2021-12-02 03:13:47,246 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-02 03:13:47,249 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 03:13:47,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:13:47,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 03:13:47,701 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:13:47,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 03:13:47,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 03:13:47,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 03:13:47,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 03:13:47,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 03:13:47,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 03:13:47,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 03:13:47,846 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-02 03:13:47,848 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.85 2021-12-02 03:13:47,848 | infra | INFO | pool-3-thread-18 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.85' started 2021-12-02 03:13:47,849 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-02 03:13:47,851 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 03:13:47,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:13:48,078 | test | DEBUG | pool-3-thread-17 | [task:_compare:2725] Comparing 0 == 0 2021-12-02 03:13:48,078 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.64': 0} for bucket 'GleamBookUsers0' 2021-12-02 03:13:48,078 | infra | INFO | pool-3-thread-17 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.24' completed 2021-12-02 03:13:48,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 03:13:48,334 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:13:48,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 03:13:48,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 03:13:48,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 03:13:48,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 03:13:48,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 03:13:48,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 03:13:48,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 03:13:48,457 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-02 03:13:48,457 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443628.46 2021-12-02 03:13:48,459 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.24 2021-12-02 03:13:48,459 | infra | INFO | pool-3-thread-19 | [task:start_task:89] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443628.46' started 2021-12-02 03:13:48,460 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 03:13:48,460 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-02 03:13:48,460 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.85 2021-12-02 03:13:48,625 | test | DEBUG | pool-3-thread-18 | [task:_compare:2725] Comparing 0 == 0 2021-12-02 03:13:48,625 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.65': 0} for bucket 'GleamBookUsers0' 2021-12-02 03:13:48,627 | infra | INFO | pool-3-thread-18 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443627.85' completed 2021-12-02 03:13:48,627 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 03:13:48,628 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443628.46 2021-12-02 03:13:49,240 | test | DEBUG | pool-3-thread-19 | [task:_compare:2725] Comparing 0 == 0 2021-12-02 03:13:49,240 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2714] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.66': 0} for bucket 'GleamBookUsers0' 2021-12-02 03:13:49,240 | infra | INFO | pool-3-thread-19 | [task:complete_task:104] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1638443628.46' completed 2021-12-02 03:13:49,242 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 03:13:49,279 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3946] Stats: {u'opsPerSec': 20033, u'diskFetches': 10, u'dataUsed': 7390024885860L, u'itemCount': 4000000000L, u'quotaPercentUsed': 78.80127166459958, u'storageTotals': {u'hdd': {u'usedByData': 13965692595537L, u'used': 14197589708631L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 21778220282025L}, u'ram': {u'usedByData': 253836663512L, u'used': 408488972288L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 13965692595537L, u'vbActiveNumNonResident': 3867227113L, u'memUsed': 253836663512L} 2021-12-02 03:13:49,654 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+-------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+-------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 4000000000 | 300.00 GiB | 236.40 GiB | 12.70 TiB | 3.319322175 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+-------------+ 2021-12-02 03:13:49,654 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+-------------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+-------------------+---------------+---------------+---------------+ | 4000000000 | 4000000000 | 40000000-80000000 | 0-0 | 0-0 | 0-0 | +---------------+---------------+-------------------+---------------+---------------+---------------+ 2021-12-02 03:13:50,000 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 3.319322175 2021-12-02 03:13:50,003 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 03:13:50,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:13:50,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 03:13:50,457 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:13:50,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 03:13:50,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 03:13:50,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 03:13:50,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 03:13:50,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 03:13:50,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 03:13:50,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 03:13:50,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-02 03:13:51,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-02 03:13:51,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-02 03:13:51,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-02 03:13:51,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-02 03:13:53,892 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-721 is 302 2021-12-02 03:13:53,892 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-73 is 309 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-731 is 343 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-11 is 312 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-228 is 302 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-13 is 338 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-14 is 350 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-87 is 359 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-808 is 325 2021-12-02 03:13:53,894 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-304 is 302 2021-12-02 03:13:53,895 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-305 is 313 2021-12-02 03:13:53,895 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-161 is 310 2021-12-02 03:13:53,895 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-89 is 309 2021-12-02 03:13:53,897 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-362 is 326 2021-12-02 03:13:53,897 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-290 is 302 2021-12-02 03:13:53,897 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-310 is 314 2021-12-02 03:13:53,897 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-25 is 311 2021-12-02 03:13:53,897 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-458 is 323 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-171 is 303 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-27 is 312 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-244 is 316 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-29 is 300 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-219 is 322 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-750 is 367 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-391 is 329 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-176 is 342 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-321 is 310 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-178 is 352 2021-12-02 03:13:53,898 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-468 is 316 2021-12-02 03:13:53,900 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-758 is 321 2021-12-02 03:13:53,900 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-148 is 346 2021-12-02 03:13:53,900 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-185 is 309 2021-12-02 03:13:53,900 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-836 is 318 2021-12-02 03:13:53,900 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-261 is 339 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-725 is 372 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-77 is 304 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-194 is 300 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-122 is 358 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-268 is 312 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-124 is 312 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-847 is 315 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-55 is 301 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-416 is 307 2021-12-02 03:13:53,901 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-275 is 331 2021-12-02 03:13:53,903 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-134 is 311 2021-12-02 03:13:53,903 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-281 is 350 2021-12-02 03:13:53,903 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-786 is 303 2021-12-02 03:13:53,903 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-787 is 322 2021-12-02 03:13:53,903 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-211 is 324 2021-12-02 03:13:53,903 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-501 is 365 2021-12-02 03:13:53,904 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-141 is 328 2021-12-02 03:13:53,904 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-151 is 322 2021-12-02 03:13:53,904 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-71 is 339 2021-12-02 03:13:53,904 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-224 is 335 2021-12-02 03:13:53,904 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 03:13:53,907 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 03:13:54,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:13:54,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 03:13:54,351 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:13:54,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 03:13:54,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 03:13:54,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 03:13:54,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 03:13:54,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 03:13:54,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 03:13:54,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 03:13:54,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-02 03:13:54,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-02 03:13:55,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-02 03:13:55,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-02 03:13:55,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-02 03:13:57,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-432 is 301 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-587 is 349 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-155 is 310 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-588 is 376 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-159 is 313 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-953 is 300 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-17 is 317 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-667 is 320 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-650 is 318 2021-12-02 03:13:57,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-20 is 306 2021-12-02 03:13:57,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-532 is 305 2021-12-02 03:13:57,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-389 is 302 2021-12-02 03:13:57,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-462 is 319 2021-12-02 03:13:57,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-31 is 302 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.33/kvstore-609 is 310 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-471 is 375 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.42/kvstore-978 is 306 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-547 is 300 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-44 is 312 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-477 is 319 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-405 is 314 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-119 is 301 2021-12-02 03:13:57,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-984 is 323 2021-12-02 03:13:57,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-409 is 321 2021-12-02 03:13:57,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-988 is 302 2021-12-02 03:13:57,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-556 is 326 2021-12-02 03:13:57,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-412 is 307 2021-12-02 03:13:57,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-52 is 350 2021-12-02 03:13:57,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-414 is 301 2021-12-02 03:13:57,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-126 is 305 2021-12-02 03:13:57,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-919 is 310 2021-12-02 03:13:57,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-487 is 313 2021-12-02 03:13:57,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-489 is 301 2021-12-02 03:13:57,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-922 is 315 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-419 is 326 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-131 is 345 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-78 is 337 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-925 is 327 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-428 is 307 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-68 is 304 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-141 is 331 2021-12-02 03:13:57,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-441 is 358 2021-12-02 03:13:57,730 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 03:13:57,733 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 03:13:57,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:13:58,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 03:13:58,196 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:13:58,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 03:13:58,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 03:13:58,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 03:13:58,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 03:13:58,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 03:13:58,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 03:13:58,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 03:13:58,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-02 03:13:58,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-02 03:13:58,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-02 03:13:59,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-02 03:13:59,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-02 03:14:01,446 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.1/kvstore-289 is 304 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-300 is 319 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.13/kvstore-517 is 303 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-230 is 370 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.15/kvstore-1023 is 323 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.2/kvstore-218 is 302 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-668 is 318 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.22/kvstore-670 is 325 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.22/kvstore-310 is 400 2021-12-02 03:14:01,447 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.27/kvstore-963 is 305 2021-12-02 03:14:01,448 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-753 is 311 2021-12-02 03:14:01,448 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-322 is 305 2021-12-02 03:14:01,448 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-683 is 302 2021-12-02 03:14:01,448 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-179 is 332 2021-12-02 03:14:01,448 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-686 is 303 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-182 is 306 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.4/kvstore-724 is 302 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.44/kvstore-764 is 304 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-981 is 321 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.46/kvstore-550 is 323 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.46/kvstore-334 is 306 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-913 is 302 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-553 is 305 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-988 is 314 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.57/kvstore-921 is 333 2021-12-02 03:14:01,450 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-929 is 304 2021-12-02 03:14:01,451 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-656 is 337 2021-12-02 03:14:01,451 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.9/kvstore-225 is 363 2021-12-02 03:14:01,451 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 03:14:01,451 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 13352424MB 2021-12-02 03:14:01,451 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 6403MB 2021-12-02 03:14:01,453 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 379303MB 2021-12-02 03:14:01,453 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 12972713MB 2021-12-02 03:14:01,456 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 03:14:01,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:14:01,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 03:14:01,900 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:14:01,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 03:14:01,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 03:14:01,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 03:14:01,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 03:14:01,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 03:14:02,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 03:14:02,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 03:14:02,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-02 03:14:02,121 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-02 03:14:02,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-02 03:14:02,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-02 03:14:03,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-02 03:14:03,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-02 03:14:04,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-02 03:14:04,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-02 03:14:04,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-02 03:14:05,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-02 03:14:05,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-02 03:14:06,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-02 03:14:06,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-02 03:14:07,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-02 03:14:07,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-02 03:14:08,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-02 03:14:08,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-02 03:14:08,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-02 03:14:09,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-02 03:14:09,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-02 03:14:10,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-02 03:14:10,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-02 03:14:10,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-02 03:14:11,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-02 03:14:11,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-02 03:14:12,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-02 03:14:12,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-02 03:14:13,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-02 03:14:13,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-02 03:14:14,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-02 03:14:14,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-02 03:14:15,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-02 03:14:15,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-02 03:14:16,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-02 03:14:16,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-02 03:14:16,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-02 03:14:17,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-02 03:14:17,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-02 03:14:18,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-02 03:14:18,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-02 03:14:19,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-02 03:14:19,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-02 03:14:20,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-02 03:14:20,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-02 03:14:21,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-02 03:14:21,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-02 03:14:21,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-02 03:14:22,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-02 03:14:22,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-02 03:14:23,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-02 03:14:23,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-02 03:14:24,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-02 03:14:24,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-02 03:14:24,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-02 03:14:25,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-02 03:14:25,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-02 03:14:26,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-02 03:14:26,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-02 03:14:27,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-02 03:14:27,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-02 03:14:28,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-02 03:14:28,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-02 03:14:29,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-02 03:14:29,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-02 03:14:30,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-02 03:14:30,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-02 03:14:30,816 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 03:14:30,819 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 03:14:30,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:14:31,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 03:14:31,289 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:14:31,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 03:14:31,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 03:14:31,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 03:14:31,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 03:14:31,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 03:14:31,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 03:14:31,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 03:14:31,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-02 03:14:31,532 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-02 03:14:31,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-02 03:14:32,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-02 03:14:32,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-02 03:14:32,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-02 03:14:33,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-02 03:14:33,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-02 03:14:34,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-02 03:14:34,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-02 03:14:35,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-02 03:14:35,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-02 03:14:36,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-02 03:14:36,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-02 03:14:37,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-02 03:14:37,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-02 03:14:38,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-02 03:14:38,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-02 03:14:38,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-02 03:14:39,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-02 03:14:39,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-02 03:14:40,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-02 03:14:40,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-02 03:14:41,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-02 03:14:41,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-02 03:14:42,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-02 03:14:42,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-02 03:14:43,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-02 03:14:43,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-02 03:14:43,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-02 03:14:44,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-02 03:14:44,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-02 03:14:45,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-02 03:14:45,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-02 03:14:46,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-02 03:14:46,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-02 03:14:46,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-02 03:14:47,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-02 03:14:47,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-02 03:14:48,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-02 03:14:48,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-02 03:14:49,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-02 03:14:49,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-02 03:14:50,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-02 03:14:50,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-02 03:14:51,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-02 03:14:51,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-02 03:14:52,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-02 03:14:52,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-02 03:14:53,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-02 03:14:53,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-02 03:14:54,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-02 03:14:54,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-02 03:14:55,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-02 03:14:55,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-02 03:14:55,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-02 03:14:56,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-02 03:14:56,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-02 03:14:57,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-02 03:14:57,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-02 03:14:58,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-02 03:14:58,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-02 03:14:59,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-02 03:14:59,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-02 03:15:00,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-02 03:15:00,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-02 03:15:00,953 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 03:15:00,956 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 03:15:01,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:15:01,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 03:15:01,433 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:15:01,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 03:15:01,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 03:15:01,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 03:15:01,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 03:15:01,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 03:15:01,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 03:15:01,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 03:15:01,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-02 03:15:01,686 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-02 03:15:01,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-02 03:15:02,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-02 03:15:02,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-02 03:15:03,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-02 03:15:03,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-02 03:15:03,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-02 03:15:04,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-02 03:15:04,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-02 03:15:05,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-02 03:15:05,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-02 03:15:06,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-02 03:15:06,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-02 03:15:07,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-02 03:15:07,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-02 03:15:07,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-02 03:15:08,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-02 03:15:08,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-02 03:15:09,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-02 03:15:09,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-02 03:15:10,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-02 03:15:10,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-02 03:15:11,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-02 03:15:11,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-02 03:15:12,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-02 03:15:12,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-02 03:15:13,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-02 03:15:13,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-02 03:15:14,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-02 03:15:14,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-02 03:15:15,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-02 03:15:15,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-02 03:15:16,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-02 03:15:16,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-02 03:15:17,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-02 03:15:17,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-02 03:15:17,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-02 03:15:18,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-02 03:15:18,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-02 03:15:19,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-02 03:15:19,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-02 03:15:20,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-02 03:15:20,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-02 03:15:21,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-02 03:15:21,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-02 03:15:22,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-02 03:15:22,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-02 03:15:23,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-02 03:15:23,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-02 03:15:24,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-02 03:15:24,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-02 03:15:25,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-02 03:15:25,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-02 03:15:25,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-02 03:15:26,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-02 03:15:26,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-02 03:15:27,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-02 03:15:27,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-02 03:15:28,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-02 03:15:28,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-02 03:15:29,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-02 03:15:29,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-02 03:15:30,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-02 03:15:30,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-02 03:15:31,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-02 03:15:31,509 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 03:15:31,509 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 29510.5, u'ReplayWALOffset': u'8625:9383936', u'NTablesCreated': 779953, u'RecentReadIOAmpSet': 0.04051939770579338, u'ReadIOAmp': 1789.48388671875, u'BlockCacheMisses': 55723674, u'NFlushes': 186271, u'RecentWriteAmp': 2.6661572456359863, u'NumSyncFlushes': 1882, u'TotalMemUsed': 132578490, u'NBloomFilterHits': 971995710, u'Fragmentation': 0.4678795337677002, u'NumNormalFlushes': 101732, u'RecentBlockCacheHitRatio': 96.22058868408203, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 419388, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 3187736, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 579341508, u'CheckpointOverhead': 7061504, u'NItemsGet': 0, u'NWriteIOs': 637823, u'NReadBytes': 33171057492L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 20835633, u'SizeRatio': 0.4516001831401478, u'NWriteBytes': 1329532928, u'BloomFilterMemUsed': 0, u'NumTables': 75, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 876544}, {u'TargetSize': 46137344, u'LogicalSize': 21637391, u'SizeRatio': 0.4689778197895397, u'NWriteBytes': 1142763520, u'BloomFilterMemUsed': 0, u'NumTables': 18, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 303104}, {u'TargetSize': 712485495, u'LogicalSize': 599007819, u'SizeRatio': 0.840729843910717, u'NWriteBytes': 4069081088L, u'BloomFilterMemUsed': 0, u'NumTables': 215, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 7024640}, {u'TargetSize': 36270067930L, u'LogicalSize': 34591615752L, u'SizeRatio': 0.953723489538554, u'NWriteBytes': 6061010944L, u'BloomFilterMemUsed': 0, u'NumTables': 783, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 342110208}, {u'TargetSize': 142497100002L, u'LogicalSize': 72540135861L, u'SizeRatio': 0.5090639448801546, u'NWriteBytes': 95814594560L, u'BloomFilterMemUsed': 0, u'NumTables': 253, u'WastedSpace': 1291585860, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 72540135424L}], u'NItemsGarbageCollected': 19275062, u'NInternalFragmentationCompacts': 44, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 444909502, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 150720, u'NNonL0LocalCompacts': 28, u'NWriterCompacts': 0, u'NReadIOs': 11277433, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 442740, u'CompactWriter': {u'NDataBlockWrites': 4378925, u'NWriteBytes': 13050302464L, u'NIndexBlockWrites': 220240, u'NBytesIncoming': 17878319148L, u'NSyncs': 0, u'NInserts': 683924198, u'NWriteIOs': 208882}, u'NWriteBytes': 108319333843L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 19275062, u'LogicalDiskSize': 72540135861L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.8696396090740227, u'BufferMemUsed': 502852, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 37948520109L, u'NTableFiles': 1515, u'NFileCountCompacts': 28, u'TableObjectMemUsed': 389360, u'TreeSnapshotMemoryUsed': 181824, u'TotalDiskUsage': 94682193920L, u'BytesIncoming': 124556578050L, u'NItemsDel': 0, u'PendingCheckpointBytes': 3960832, u'NCompacts': 59957, u'NItemsPut': 128654800, u'NSyncs': 84539, u'CompactReader': {u'NReadIOs': 11277433, u'NReadBytes': 33171057492L, u'NDataBlockReads': 10957406, u'NBytesOutgoing': 0, u'NIndexBlockReads': 320027}, u'ActiveDiskUsage': 72890449920L, u'NTablesDeleted': 257254, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 28906525, u'NWriteBytes': 95269031379L, u'NIndexBlockWrites': 694448, u'NBytesIncoming': 125711676935L, u'NSyncs': 84539, u'NInserts': 128651016, u'NWriteIOs': 428941}}, u'NReadBytes': 93176587073593L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 39.56666564941406, u'NInternalFragmentationCompacts': 44, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 471542528, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 8192, u'NTxns': 7170775, u'NWriteBytes': 145858854912L, u'NSegments': 8625, u'NSyncs': 7170775, u'NLoggedBytes': 145858854912L, u'NActiveSegments': 1, u'NWrites': 7194594, u'NInserts': 149859644, u'DiskUsed': 15843328}, u'NGets': 19497058, u'NReadBytesSet': 81874269163L, u'NReadBytesCompact': 95274558710L, u'IndexResidentRatio': 0.03880370780825615, u'CompactQueueSize': 0, u'NSyncsPerSec': 1.100000023841858, u'BlockCacheHitRatio': 72.58677673339844, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 93081312192465L, u'BloomFilterMemUsedRatio': 0.27977877855300903, u'NWriteBytes': 322436812447L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.0, u'WriteAmp': 2.5339691638946533, u'NBlocksCached': 29252, u'LogicalDataSize': 39523728183L, u'ActiveBloomFilterMemUsed': 41652448, u'NFileCountCompacts': 19044, u'NTableFiles': 2367, u'FlushQueueSize': 72, u'TableObjectMemUsed': 600656, u'BlockCacheQuota': 84914312, u'TotalDiskUsage': 96660115456L, u'NPurgerRuns': 165, u'ReadIOAmpSet': 0.0, u'NSyncs': 253617, u'NReadIOsSetPerSec': 8.633333206176758, u'NReadIOsPerSec': 61.36666488647461, u'NTablesDeleted': 617260, u'BloomFilterFPR': 0.015472772531211376, u'NBloomFilterHitsPerSec': 1704.2332763671875, u'WriteCacheMemUsed': 4402248, u'RecentBloomFilterFPR': 0.01586109958589077, u'BytesPerRead': 4769921.5, u'BytesIncomingPerSec': 389870.28125, u'NReadIOsGetPerSec': 11.366666793823242, u'NGetStats': 454967, u'TotalIndexBlocksSize': 606484151, u'NGetStatsPerSec': 2.9666666984558105, u'NWriteBytesCompact': 73088471040L, u'NReadIOsSet': 30751962, u'ActiveDataSize': 39523728183L, u'NWriteBytesCompactPerSec': 127112.53125, u'NBloomFilterMissesPerSec': 27.46666717529297, u'NCommitBatches': 7170775, u'NGetStatsComputed': 454967, u'NWriteIOs': 8467496, u'NReadIOsGet': 34920423232L, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 21228578, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 213.06666564941406, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.7333333492279053, u'ManagedFileObjectMemUsed': 232512, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 83599645, u'NWriterCompacts': 0, u'NReadIOs': 34955141501L, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 1442938347, u'NTablesCreated': 189539, u'ReadIOAmp': 331.7313335804761, u'MemtablesMemUsed': 189464, u'NBloomFilterHits': 971995710, u'TotalIndexBlocksSize': 27130403, u'CheckpointOverhead': 35123200, u'NItemsGet': 105267108, u'NWriteIOs': 379514, u'NReadBytes': 93143210197096L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 2090929, u'SizeRatio': 0.04531966556202282, u'NWriteBytes': 6725984256L, u'BloomFilterMemUsed': 28224, u'NumTables': 94, u'WastedSpace': 134, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1986560}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 33479246, u'SizeRatio': 0.7256431146101519, u'NWriteBytes': 24813813760L, u'BloomFilterMemUsed': 1039712, u'NumTables': 61, u'WastedSpace': 58963, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 33464320}, {u'TargetSize': 173564364, u'LogicalSize': 160560748, u'SizeRatio': 0.9250789983593637, u'NWriteBytes': 13411282944L, u'BloomFilterMemUsed': 4899840, u'NumTables': 81, u'WastedSpace': 442231, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 160997376}, {u'TargetSize': 1735643687, u'LogicalSize': 1735643687, u'SizeRatio': 1.0, u'NWriteBytes': 22220365824L, u'BloomFilterMemUsed': 35683328, u'NumTables': 232, u'WastedSpace': 6105267, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1741746176}], u'NItemsGarbageCollected': 38125569, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 26628526, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 57792, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 34943497085L, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 182953, u'CompactWriter': {u'NDataBlockWrites': 21960841, u'NWriteBytes': 59960369152L, u'NIndexBlockWrites': 282854, u'NBytesIncoming': 83690334126L, u'NSyncs': 0, u'NInserts': 1442938347, u'NWriteIOs': 133117}, u'NWriteBytes': 66814208929L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 34920398119L, u'NReadBytes': 93081276298878L, u'NDataBlockReads': 66261835280L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 130000018657L}, u'NBloomFilterMisses': 15275828, u'NItemsMerged': 0, u'LogicalDiskSize': 1735643687, u'TotalBloomFilterMemUsed': 41722088, u'WriteAmp': 15.89781254050195, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 41651104, u'LogicalDataSize': 1575082939, u'NTableFiles': 602, u'NFileCountCompacts': 9519, u'TableObjectMemUsed': 149296, u'TreeSnapshotMemoryUsed': 95488, u'TotalDiskUsage': 1975873536, u'BytesIncoming': 4202729700L, u'NItemsDel': 0, u'PendingCheckpointBytes': 4878336, u'NCompacts': 12547, u'NItemsPut': 85770050, u'NSyncs': 84539, u'CompactReader': {u'NReadIOs': 23098798, u'NReadBytes': 61933575800L, u'NDataBlockReads': 22662336, u'NBytesOutgoing': 0, u'NIndexBlockReads': 436462}, u'ActiveDiskUsage': 1938194432, u'NTablesDeleted': 189071, u'IterateReader': {u'NReadIOs': 168, u'NReadBytes': 322418, u'NDataBlockReads': 84, u'NBytesOutgoing': 0, u'NIndexBlockReads': 84}, u'BloomFilterFPR': 0.015472772598048906, u'Writer': {u'NDataBlockWrites': 1423750, u'NWriteBytes': 6853839777L, u'NIndexBlockWrites': 162292, u'NBytesIncoming': 4974550414L, u'NSyncs': 84539, u'NInserts': 85768158, u'NWriteIOs': 246397}}, u'NReadBytesCompactPerSec': 126597.46875, u'TableMetaMemUsed': 697433, u'NDeletes': 0, u'BlockCacheHitsPerSec': 1007.3333129882812, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 15275828, u'LogicalDiskSize': 74275904683L, u'NSets': 85770050, u'TotalBloomFilterMemUsed': 41723632, u'BufferMemUsed': 502852, u'RecentReadAmp': 0.0, u'NGetsPerSec': 0.0, u'NInserts': 0, u'NCommitBatchesPerSec': 158.3333282470703, u'FragmentationWithOverhead': 0.5911061763763428, u'NCompactsPerSec': 0.13333334028720856, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 156107.96875, u'TreeSnapshotMemoryUsed': 322320, u'BytesIncoming': 127245762708L, u'NCompacts': 82001, u'NumPersistentFlushes': 82657, u'NWriteIOsPerSec': 162.93333435058594, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 0.0, u'ActiveDiskUsage': 74829389824L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 2.9666666984558105, u'NReadBytesSetPerSec': 22732.56640625, u'BlockCacheHits': 147549275, u'NWriteBytesPerSec': 1039455.4375, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 171026, u'ReadIOAmp': 0.0017479438881683417, u'MemtablesMemUsed': 1025048, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 12240, u'CheckpointOverhead': 1155072, u'NItemsGet': 14367166, u'NWriteIOs': 255565, u'NReadBytes': 205819005, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 725071, u'SizeRatio': 0.015715490687977184, u'NWriteBytes': 1323290624, u'BloomFilterMemUsed': 640, u'NumTables': 80, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 655360}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 125135, u'SizeRatio': 0.002712228081443093, u'NWriteBytes': 77799424, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 42573534, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 4500, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 24000, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 366983, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 71740, u'CompactWriter': {u'NDataBlockWrites': 9497, u'NWriteBytes': 77799424, u'NIndexBlockWrites': 9497, u'NBytesIncoming': 43175061, u'NSyncs': 0, u'NInserts': 503341, u'NWriteIOs': 9497}, u'NWriteBytes': 1444414763, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 25113, u'NReadBytes': 35893587, u'NDataBlockReads': 7159646, u'NBytesOutgoing': 0, u'NIndexBlockReads': 7159646}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 125135, u'TotalBloomFilterMemUsed': 1544, u'WriteAmp': 0.33780154813157187, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1344, u'LogicalDataSize': 125135, u'NTableFiles': 250, u'NFileCountCompacts': 9497, u'TableObjectMemUsed': 62000, u'TreeSnapshotMemoryUsed': 45008, u'TotalDiskUsage': 2048000, u'BytesIncoming': 4275927008L, u'NItemsDel': 0, u'PendingCheckpointBytes': 155648, u'NCompacts': 9497, u'NItemsPut': 42577283, u'NSyncs': 84539, u'CompactReader': {u'NReadIOs': 341870, u'NReadBytes': 169925418, u'NDataBlockReads': 170935, u'NBytesOutgoing': 0, u'NIndexBlockReads': 170935}, u'ActiveDiskUsage': 745472, u'NTablesDeleted': 170935, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 161529, u'NWriteBytes': 1366615339, u'NIndexBlockWrites': 161529, u'NBytesIncoming': 120786928, u'NSyncs': 84539, u'NInserts': 1575979, u'NWriteIOs': 246068}}}}} 2021-12-02 03:15:31,513 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.4678795337677002, 0.4704144597053528, 0.4684290587902069, 0.4738217890262604, 0.46076223254203796, 0.4669061303138733, 0.4657285809516907, 0.46527159214019775, 0.473688542842865, 0.46979084610939026, 0.47239893674850464, 0.4682061970233917, 0.47068697214126587, 0.4619098901748657, 0.46349114179611206, 0.471623957157135, 0.46454066038131714, 0.4611978530883789, 0.46267881989479065, 0.46623316407203674, 0.4673308730125427, 0.4708329439163208, 0.4685091972351074, 0.4638839364051819, 0.46339550614356995, 0.4659776985645294, 0.46676719188690186, 0.4720921516418457, 0.4627762734889984, 0.4611952006816864, 0.4559684097766876, 0.4639139771461487, 0.4596552848815918, 0.45865318179130554, 0.4613683521747589, 0.46210813522338867, 0.46133625507354736, 0.4548838138580322, 0.4564320147037506, 0.4591260850429535, 0.4616096615791321, 0.45816770195961, 0.46023547649383545, 0.45866188406944275, 0.4615785479545593, 0.4585941433906555, 0.4642326533794403, 0.45499342679977417, 0.45828911662101746, 0.4530748426914215, 0.4576333463191986, 0.4608323574066162, 0.4509941041469574, 0.4586050510406494, 0.46922728419303894, 0.45939791202545166, 0.46666958928108215, 0.4600003957748413, 0.45495346188545227, 0.4447706937789917, 0.4628749191761017, 0.46822527050971985, 0.4711731970310211, 0.46755126118659973], '172.23.110.66': [0.46864330768585205, 0.4626341164112091, 0.47323089838027954, 0.4662763476371765, 0.46691837906837463, 0.46995052695274353, 0.45669230818748474, 0.4642087519168854, 0.46639442443847656, 0.4712807536125183, 0.46956440806388855, 0.46940019726753235, 0.4661772549152374, 0.47298118472099304, 0.4672324061393738, 0.47057777643203735, 0.46766337752342224, 0.47070252895355225, 0.46402496099472046, 0.4681248664855957, 0.4663366377353668, 0.47136950492858887, 0.47113168239593506, 0.46848592162132263, 0.4700716435909271, 0.46179670095443726, 0.4727860689163208, 0.4728156626224518, 0.4725644588470459, 0.47081732749938965, 0.4786461591720581, 0.4681923985481262, 0.4748392701148987, 0.46772128343582153, 0.4741034507751465, 0.4774235486984253, 0.4777619540691376, 0.4664112329483032, 0.4755721390247345, 0.4734521806240082, 0.4667011499404907, 0.4697214961051941, 0.4682369828224182, 0.4728599488735199, 0.4679756164550781, 0.4675271511077881, 0.4705016613006592, 0.4722340703010559, 0.46222537755966187, 0.47530847787857056, 0.46649491786956787, 0.4591098725795746, 0.4695936143398285, 0.47504720091819763, 0.4780186116695404, 0.4690927565097809, 0.46506184339523315, 0.4706578850746155, 0.4645273685455322, 0.4712585210800171, 0.4632778763771057, 0.47045019268989563, 0.4673038125038147, 0.4642746150493622], '172.23.110.64': [0.4626557528972626, 0.4712734520435333, 0.4614785611629486, 0.45822304487228394, 0.45755359530448914, 0.4582718312740326, 0.4649643898010254, 0.4602403938770294, 0.46096575260162354, 0.4614753723144531, 0.4683327078819275, 0.4635295271873474, 0.4565392732620239, 0.46480593085289, 0.4623485505580902, 0.46687933802604675, 0.45048174262046814, 0.46289899945259094, 0.4614242911338806, 0.4626357853412628, 0.45815804600715637, 0.46372970938682556, 0.46826374530792236, 0.45391330122947693, 0.45515763759613037, 0.46554455161094666, 0.46476975083351135, 0.467601478099823, 0.4614545404911041, 0.45472997426986694, 0.4645775258541107, 0.46015235781669617, 0.4716290831565857, 0.46474897861480713, 0.46299102902412415, 0.4674052298069, 0.46790823340415955, 0.45965439081192017, 0.46964704990386963, 0.47301429510116577, 0.46939438581466675, 0.4710654020309448, 0.46850088238716125, 0.46558284759521484, 0.4567161500453949, 0.45370209217071533, 0.47107475996017456, 0.47131282091140747, 0.4629392623901367, 0.4658852815628052, 0.4540550112724304, 0.46379509568214417, 0.47148898243904114, 0.4690966308116913, 0.4668698310852051, 0.4587341547012329, 0.4672911763191223, 0.4608316421508789, 0.46135008335113525, 0.46254628896713257, 0.4694972336292267, 0.46748223900794983, 0.4599528908729553, 0.46312496066093445]} 2021-12-02 03:15:31,845 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 47.0 2021-12-02 03:15:32,176 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 47.0 2021-12-02 03:15:32,516 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 47.0 2021-12-02 03:15:32,516 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 47.0, '172.23.110.66': 47.0, '172.23.110.64': 47.0} 2021-12-02 03:15:32,519 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 03:15:32,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:15:32,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 03:15:32,963 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:15:32,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 03:15:32,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 03:15:33,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 03:15:33,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 03:15:33,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 03:15:33,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 03:15:33,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 03:15:33,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 03:15:33,125 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 03:15:33,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 03:15:33,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:33,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:33,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:15:33,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:15:33,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:33,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:33,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:15:33,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:15:33,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:33,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:33,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:15:33,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:15:33,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:33,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:33,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 03:15:33,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 03:15:33,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 03:15:33,884 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 03:15:33,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 03:15:33,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:33,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 03:15:33,953 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 03:15:33,953 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 03:15:33,956 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 03:15:34,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:15:34,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 03:15:34,387 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:15:34,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 03:15:34,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 03:15:34,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 03:15:34,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 03:15:34,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 03:15:34,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 03:15:34,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 03:15:34,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 03:15:34,542 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 03:15:34,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 03:15:34,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:34,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:34,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:15:34,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:15:34,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:34,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:34,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:15:34,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:15:34,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:34,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:35,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:15:35,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:15:35,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 03:15:35,073 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 03:15:35,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 03:15:35,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:35,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 03:15:35,118 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 03:15:35,118 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 03:15:35,121 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 03:15:35,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:15:35,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 03:15:35,595 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:15:35,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 03:15:35,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 03:15:35,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 03:15:35,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 03:15:35,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 03:15:35,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 03:15:35,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 03:15:35,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 03:15:35,749 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 03:15:35,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 03:15:35,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:35,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:36,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:15:36,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:15:36,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:36,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:36,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:15:36,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:15:36,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:36,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:15:36,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:15:36,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:15:36,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 03:15:36,335 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 03:15:36,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 03:15:36,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:15:36,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 03:15:36,378 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 03:15:36,378 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 ############################################################ # # RAM FootPrint: 7005.04296875 # ############################################################ Read Start: 0 Read End: 40000000 Update Start: 80000000 Update End: 800000000 Expiry Start: 0 Expiry End: 0 Delete Start: 40000000 Delete End: 80000000 Create Start: 80000000 Create End: 800000000 Final Start: 80000000 Final End: 800000000 2021-12-02 03:15:36,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:36,426 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1638443737.43 2021-12-02 03:15:37,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:37,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1638443738.44 2021-12-02 03:15:38,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:38,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1638443739.45 2021-12-02 03:15:39,456 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:39,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1638443740.46 2021-12-02 03:15:40,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:40,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1638443741.47 2021-12-02 03:15:41,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:41,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1638443742.48 2021-12-02 03:15:42,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:42,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1638443743.51 2021-12-02 03:15:43,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:43,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1638443744.52 2021-12-02 03:15:44,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:44,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1638443745.53 2021-12-02 03:15:45,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:45,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1638443746.54 2021-12-02 03:15:46,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:46,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:47,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:47,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:48,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:48,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:49,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:49,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:50,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:50,619 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:51,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:51,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:52,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:52,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:53,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:53,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:54,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:54,670 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:55,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:55,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:56,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:56,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:57,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:57,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:58,719 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:58,719 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:15:59,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:15:59,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:00,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:00,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:01,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:01,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:02,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:02,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:03,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:03,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:04,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:04,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:05,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:05,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:06,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:06,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:07,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:07,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:08,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:08,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:09,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:09,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:10,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:10,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:11,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:11,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:12,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:12,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:13,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:13,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:14,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:14,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:15,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:15,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:16,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:16,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:17,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:17,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:18,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:18,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:19,915 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:19,915 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:20,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:20,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:21,933 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:21,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:22,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:22,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:23,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:23,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:24,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:24,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-02 03:16:25,964 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-02 03:16:25,966 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None ############################################################ # # Step 4: Rebalance in with Loading of docs # ############################################################ KV nodes in cluster: ['172.23.110.65', '172.23.110.66'] CBAS nodes in cluster: [] INDEX nodes in cluster: [] FTS nodes in cluster: [] QUERY nodes in cluster: [] EVENTING nodes in cluster: [] AVAILABLE nodes for cluster: ['172.23.110.67', '172.23.110.68'] Servers coming in : ['172.23.110.68'] with services: ['kv'] Servers going out : [] NEW AVAILABLE nodes for cluster: ['172.23.110.67'] 2021-12-02 03:16:26,993 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.110.68]_OUT=[]_1638443786.97 2021-12-02 03:16:26,993 | infra | INFO | pool-3-thread-20 | [task:start_task:89] Thread 'Rebalance_task_IN=[172.23.110.68]_OUT=[]_1638443786.97' started 2021-12-02 03:16:26,996 | test | DEBUG | pool-3-thread-20 | [rest_client:add_node:1178] Adding remote node 172.23.110.68:8091 to cluster 172.23.110.64:8091 2021-12-02 03:16:27,002 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L466 2021-12-02 03:16:27,002 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep for 60s for rebalance to start 2021-12-02 03:16:31,309 | test | DEBUG | pool-3-thread-20 | [rest_client:rebalance:1431] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-02 03:16:31,364 | test | DEBUG | pool-3-thread-20 | [rest_client:rebalance:1436] Rebalance operation started 2021-12-02 03:16:31,364 | test | INFO | pool-3-thread-20 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1817-enterprise | 10.5978222887 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1817-enterprise | 12.2151615251 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1817-enterprise | 12.8767008349 | Cluster node | | 172.23.110.68 | ['kv'] | | | <--- IN --- | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-02 03:16:31,372 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:16:31,372 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:16:31,385 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:16:31,388 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:16:31,388 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:16:41,403 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:16:41,403 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:16:41,418 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:16:41,421 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:16:41,421 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:16:51,434 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:16:51,434 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:16:51,448 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:16:51,451 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:16:51,453 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:01,466 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:01,466 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:17:01,486 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:17:01,489 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:17:01,489 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:11,516 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:11,516 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:17:11,536 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:17:11,539 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:17:11,539 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:21,553 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:21,555 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:17:21,572 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:17:21,575 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:17:21,575 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:27,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L765 2021-12-02 03:17:27,007 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 30 seconds. Reason: Let the rebalance begin! 2021-12-02 03:17:31,588 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:31,589 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:17:31,611 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:17:31,615 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:17:31,615 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:41,651 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:41,651 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:17:41,668 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:17:41,673 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:17:41,673 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:51,687 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:51,687 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 03:17:51,703 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:17:51,707 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:17:51,707 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:17:57,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 03:17:57,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:17:57,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:01,720 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.19 % 2021-12-02 03:18:01,720 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.19 2021-12-02 03:18:01,734 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:18:01,737 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:18:01,737 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:07,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.39 % 2021-12-02 03:18:07,061 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:18:07,061 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:11,752 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:11,752 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:18:11,769 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:18:11,772 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:18:11,772 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:17,076 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:17,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:18:17,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:21,783 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:21,785 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:18:21,799 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:18:21,802 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:18:21,802 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:27,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:27,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:18:27,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:31,813 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:31,813 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:18:31,828 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:18:31,829 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:18:31,829 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:37,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:37,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:18:37,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:41,842 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:41,842 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:18:41,858 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:18:41,859 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:18:41,859 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:47,132 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:47,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:18:47,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:51,871 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:51,871 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:18:51,888 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:18:51,891 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:18:51,891 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:18:57,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:18:57,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:18:57,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:01,904 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:19:01,904 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:19:01,924 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:19:01,927 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:19:01,927 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:07,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:19:07,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:19:07,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:11,940 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:19:11,940 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:19:11,954 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:19:11,957 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:19:11,957 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:17,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:19:17,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:19:17,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:21,970 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.78 % 2021-12-02 03:19:21,970 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.78 2021-12-02 03:19:21,992 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:19:21,993 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:19:21,993 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:27,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.98 % 2021-12-02 03:19:27,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:19:27,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:32,006 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.98 % 2021-12-02 03:19:32,007 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 0.98 2021-12-02 03:19:32,025 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:19:32,026 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:19:32,026 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:37,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.17 % 2021-12-02 03:19:37,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:19:37,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:42,040 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:19:42,042 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:19:42,055 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:19:42,058 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:19:42,058 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:47,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:19:47,262 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:19:47,262 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:52,072 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:19:52,072 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:19:52,092 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:19:52,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:19:52,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:19:57,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:19:57,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:19:57,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:02,108 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:02,108 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:20:02,127 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:20:02,128 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:20:02,128 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:07,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:07,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:20:07,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:12,142 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:12,142 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:20:12,161 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:20:12,164 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:20:12,164 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:17,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:17,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:20:17,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:22,177 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:22,178 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:20:22,197 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:20:22,198 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:20:22,198 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:27,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:27,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:20:27,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:32,213 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:32,213 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:20:32,229 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:20:32,230 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:20:32,230 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:37,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:37,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:20:37,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:42,249 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:42,249 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 03:20:42,269 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:20:42,272 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:20:42,272 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:47,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 03:20:47,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:20:47,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:52,289 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.76 % 2021-12-02 03:20:52,289 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.76 2021-12-02 03:20:52,302 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:20:52,305 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:20:52,305 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:20:57,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.76 % 2021-12-02 03:20:57,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:20:57,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:02,318 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.76 % 2021-12-02 03:21:02,319 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 1.76 2021-12-02 03:21:02,332 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:21:02,335 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:21:02,335 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:07,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.76 % 2021-12-02 03:21:07,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:21:07,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:12,348 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.15 % 2021-12-02 03:21:12,349 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.15 2021-12-02 03:21:12,368 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:21:12,369 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:21:12,369 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:17,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:17,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:21:17,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:22,384 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:22,384 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 03:21:22,404 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:21:22,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:21:22,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:27,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:27,471 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:21:27,471 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:32,421 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:32,421 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 03:21:32,444 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:21:32,447 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:21:32,447 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:37,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:37,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:21:37,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:42,463 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:42,463 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 03:21:42,483 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:21:42,486 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:21:42,486 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:47,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:47,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:21:47,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:52,499 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:52,500 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 03:21:52,522 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:21:52,529 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:21:52,529 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:21:57,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:21:57,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:21:57,553 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:02,546 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:22:02,546 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 03:22:02,588 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:22:02,592 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:22:02,592 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:07,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:22:07,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:22:07,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:12,605 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 03:22:12,605 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 03:22:12,625 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:22:12,631 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:22:12,631 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:17,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.54 % 2021-12-02 03:22:17,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:22:17,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:22,647 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.54 % 2021-12-02 03:22:22,647 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.54 2021-12-02 03:22:22,664 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:22:22,667 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:22:22,668 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:27,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.54 % 2021-12-02 03:22:27,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:22:27,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:32,680 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.73 % 2021-12-02 03:22:32,681 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.73 2021-12-02 03:22:32,697 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:22:32,700 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:22:32,700 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:37,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.73 % 2021-12-02 03:22:37,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:22:37,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:42,713 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.73 % 2021-12-02 03:22:42,714 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 2.73 2021-12-02 03:22:42,732 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:22:42,734 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:22:42,734 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:47,678 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.93 % 2021-12-02 03:22:47,688 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:22:47,688 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:52,747 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:22:52,747 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.13 2021-12-02 03:22:52,763 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:22:52,766 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:22:52,766 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:22:57,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:22:57,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:22:57,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:02,783 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:02,785 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.13 2021-12-02 03:23:02,799 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:23:02,803 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:23:02,803 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:07,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:07,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:23:07,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:12,816 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:12,818 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.13 2021-12-02 03:23:12,834 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:23:12,835 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:23:12,836 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:17,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:17,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:23:17,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:22,852 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:22,854 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.13 2021-12-02 03:23:22,871 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:23:22,872 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:23:22,874 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:27,783 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:27,795 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:23:27,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:32,884 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:32,884 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.13 2021-12-02 03:23:32,900 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:23:32,901 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:23:32,903 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:37,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.13 % 2021-12-02 03:23:37,816 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:23:37,816 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:42,914 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.32 % 2021-12-02 03:23:42,914 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.32 2021-12-02 03:23:42,927 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:23:42,930 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:23:42,930 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:47,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.32 % 2021-12-02 03:23:47,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:23:47,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:52,940 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.32 % 2021-12-02 03:23:52,940 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.32 2021-12-02 03:23:52,954 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:23:52,957 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:23:52,957 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:23:57,849 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.32 % 2021-12-02 03:23:57,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:23:57,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:02,967 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.52 % 2021-12-02 03:24:02,969 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.52 2021-12-02 03:24:02,983 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:24:02,986 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:24:02,986 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:07,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.52 % 2021-12-02 03:24:07,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:24:07,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:12,997 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.52 % 2021-12-02 03:24:12,997 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.52 2021-12-02 03:24:13,012 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:24:13,013 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:24:13,013 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:17,890 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:17,895 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:24:17,897 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:23,026 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:23,026 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.91 2021-12-02 03:24:23,052 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:24:23,055 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:24:23,055 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:27,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:27,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:24:27,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:33,068 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:33,068 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.91 2021-12-02 03:24:33,082 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:24:33,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:24:33,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:37,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:37,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:24:37,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:43,098 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:43,098 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.91 2021-12-02 03:24:43,117 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:24:43,118 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:24:43,119 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:47,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:47,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:24:47,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:53,131 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:53,132 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.91 2021-12-02 03:24:53,148 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:24:53,151 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:24:53,151 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:24:57,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:24:57,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:24:57,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:03,164 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.91 % 2021-12-02 03:25:03,164 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 3.91 2021-12-02 03:25:03,178 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:25:03,180 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:25:03,180 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:07,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.10 % 2021-12-02 03:25:07,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:25:07,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:13,194 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.10 % 2021-12-02 03:25:13,194 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.1 2021-12-02 03:25:13,213 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:25:13,216 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:25:13,216 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:17,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.10 % 2021-12-02 03:25:18,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:25:18,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:23,229 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.10 % 2021-12-02 03:25:23,230 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.1 2021-12-02 03:25:23,250 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:25:23,253 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:25:23,253 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:28,017 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 03:25:28,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:25:28,025 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:33,266 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 03:25:33,266 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.3 2021-12-02 03:25:33,289 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:25:33,290 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:25:33,292 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:38,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 03:25:38,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:25:38,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:43,305 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.49 % 2021-12-02 03:25:43,305 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.49 2021-12-02 03:25:43,325 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:25:43,326 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:25:43,328 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:48,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:25:48,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:25:48,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:53,341 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:25:53,341 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.69 2021-12-02 03:25:53,361 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:25:53,364 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:25:53,364 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:25:58,084 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:25:58,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:25:58,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:03,377 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:26:03,378 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.69 2021-12-02 03:26:03,398 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:26:03,400 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:26:03,400 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:08,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:26:08,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:26:08,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:13,413 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:26:13,413 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.69 2021-12-02 03:26:13,434 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:26:13,437 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:26:13,437 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:18,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:26:18,122 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:26:18,122 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:23,453 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:26:23,453 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.69 2021-12-02 03:26:23,470 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:26:23,473 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:26:23,473 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:28,135 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 03:26:28,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:26:28,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:33,493 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.88 % 2021-12-02 03:26:33,493 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.88 2021-12-02 03:26:33,513 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:26:33,517 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:26:33,517 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:38,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.88 % 2021-12-02 03:26:38,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:26:38,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:43,532 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.88 % 2021-12-02 03:26:43,532 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 4.88 2021-12-02 03:26:43,549 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:26:43,552 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:26:43,552 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:48,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.88 % 2021-12-02 03:26:48,176 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:26:48,176 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:53,565 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.08 % 2021-12-02 03:26:53,566 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.08 2021-12-02 03:26:53,588 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:26:53,589 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:26:53,589 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:26:58,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.08 % 2021-12-02 03:26:58,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:26:58,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:03,605 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.27 % 2021-12-02 03:27:03,605 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.27 2021-12-02 03:27:03,621 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:27:03,625 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:27:03,625 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:08,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.27 % 2021-12-02 03:27:08,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:27:08,216 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:13,638 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.27 % 2021-12-02 03:27:13,638 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.27 2021-12-02 03:27:13,657 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:27:13,660 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:27:13,660 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:18,229 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:18,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:27:18,234 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:23,674 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:23,674 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 03:27:23,688 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:27:23,691 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:27:23,691 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:28,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:28,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:27:28,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:33,706 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:33,706 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 03:27:33,723 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:27:33,726 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:27:33,726 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:38,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:38,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:27:38,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:43,743 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:43,743 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 03:27:43,759 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:27:43,762 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:27:43,762 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:48,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:48,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:27:48,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:53,775 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:53,776 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 03:27:53,793 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:27:53,798 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:27:53,798 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:27:58,303 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:27:58,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:27:58,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:03,812 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 03:28:03,812 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 03:28:03,828 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:28:03,832 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:28:03,832 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:08,321 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.86 % 2021-12-02 03:28:08,326 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:28:08,326 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:13,845 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.86 % 2021-12-02 03:28:13,846 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 5.86 2021-12-02 03:28:13,862 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:28:13,868 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:28:13,868 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:18,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.86 % 2021-12-02 03:28:18,344 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:28:18,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:23,880 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.06 % 2021-12-02 03:28:23,881 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.06 2021-12-02 03:28:23,897 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:28:23,900 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:28:23,900 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:28,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.06 % 2021-12-02 03:28:28,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:28:28,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:33,914 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.06 % 2021-12-02 03:28:33,914 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.06 2021-12-02 03:28:33,930 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:28:33,933 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:28:33,933 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:38,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.06 % 2021-12-02 03:28:38,385 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:28:38,385 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:43,946 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 03:28:43,946 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.25 2021-12-02 03:28:43,960 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:28:43,963 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:28:43,963 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:48,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 03:28:48,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:28:48,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:53,986 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 03:28:53,986 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.25 2021-12-02 03:28:54,002 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:28:54,003 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:28:54,003 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:28:58,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 03:28:58,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:28:58,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:04,016 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 03:29:04,016 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.25 2021-12-02 03:29:04,029 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:29:04,032 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:29:04,032 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:08,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 03:29:08,443 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:29:08,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:14,042 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.45 % 2021-12-02 03:29:14,042 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.45 2021-12-02 03:29:14,056 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:29:14,059 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:29:14,059 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:18,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.45 % 2021-12-02 03:29:18,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:29:18,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:24,069 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.45 % 2021-12-02 03:29:24,071 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.45 2021-12-02 03:29:24,084 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:29:24,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:29:24,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:28,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.45 % 2021-12-02 03:29:28,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:29:28,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:34,096 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.45 % 2021-12-02 03:29:34,098 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.45 2021-12-02 03:29:34,114 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:29:34,115 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:29:34,115 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:38,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.64 % 2021-12-02 03:29:38,510 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:29:38,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:44,127 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.64 % 2021-12-02 03:29:44,128 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.64 2021-12-02 03:29:44,141 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:29:44,144 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:29:44,144 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:48,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.64 % 2021-12-02 03:29:48,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:29:48,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:54,154 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.84 % 2021-12-02 03:29:54,154 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.84 2021-12-02 03:29:54,168 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:29:54,171 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:29:54,171 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:29:58,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.84 % 2021-12-02 03:29:58,558 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:29:58,558 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:04,183 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.84 % 2021-12-02 03:30:04,184 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 6.84 2021-12-02 03:30:04,198 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:30:04,200 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:30:04,200 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:08,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.84 % 2021-12-02 03:30:08,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:30:08,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:14,213 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 03:30:14,213 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.03 2021-12-02 03:30:14,229 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:30:14,230 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:30:14,230 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:18,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 03:30:18,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:30:18,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:24,243 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 03:30:24,243 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.03 2021-12-02 03:30:24,262 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:30:24,263 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:30:24,263 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:28,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 03:30:28,632 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:30:28,632 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:34,276 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 03:30:34,278 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.03 2021-12-02 03:30:34,296 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:30:34,299 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:30:34,299 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:38,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.23 % 2021-12-02 03:30:38,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:30:38,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:44,311 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.23 % 2021-12-02 03:30:44,311 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.23 2021-12-02 03:30:44,325 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:30:44,328 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:30:44,328 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:48,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.23 % 2021-12-02 03:30:48,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:30:48,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:54,338 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.23 % 2021-12-02 03:30:54,338 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.23 2021-12-02 03:30:54,352 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:30:54,355 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:30:54,355 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:30:58,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.23 % 2021-12-02 03:30:58,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:30:58,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:04,368 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.42 % 2021-12-02 03:31:04,368 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.42 2021-12-02 03:31:04,382 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:31:04,385 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:31:04,385 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:08,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.42 % 2021-12-02 03:31:08,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:31:08,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:14,400 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.42 % 2021-12-02 03:31:14,400 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.42 2021-12-02 03:31:14,414 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:31:14,415 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:31:14,415 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:18,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.62 % 2021-12-02 03:31:18,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:31:18,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:24,427 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.62 % 2021-12-02 03:31:24,427 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.62 2021-12-02 03:31:24,443 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:31:24,444 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:31:24,444 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:28,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.62 % 2021-12-02 03:31:28,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:31:28,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:34,457 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.62 % 2021-12-02 03:31:34,459 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.62 2021-12-02 03:31:34,479 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:31:34,480 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:31:34,480 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:38,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 03:31:38,808 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:31:38,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:44,496 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 03:31:44,496 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.81 2021-12-02 03:31:44,519 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:31:44,520 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:31:44,520 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:48,823 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 03:31:48,835 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:31:48,835 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:54,535 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 03:31:54,535 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 7.81 2021-12-02 03:31:54,555 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:31:54,558 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:31:54,558 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:31:58,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:31:58,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:31:58,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:04,571 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:04,571 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.01 2021-12-02 03:32:04,591 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:32:04,592 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:32:04,594 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:08,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:08,923 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:32:08,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:14,609 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:14,609 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.01 2021-12-02 03:32:14,631 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:32:14,634 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:32:14,634 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:18,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:18,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:32:18,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:24,651 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:24,653 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.01 2021-12-02 03:32:24,673 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:32:24,677 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:32:24,677 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:28,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:28,971 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:32:28,971 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:34,690 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 03:32:34,690 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.01 2021-12-02 03:32:34,706 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:32:34,709 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:32:34,709 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:38,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.20 % 2021-12-02 03:32:38,994 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:32:38,994 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:44,720 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.20 % 2021-12-02 03:32:44,720 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.2 2021-12-02 03:32:44,737 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:32:44,740 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:32:44,740 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:49,010 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.20 % 2021-12-02 03:32:49,020 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:32:49,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:54,752 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.40 % 2021-12-02 03:32:54,752 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.4 2021-12-02 03:32:54,766 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:32:54,767 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:32:54,769 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:32:59,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.40 % 2021-12-02 03:32:59,040 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:32:59,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:04,782 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.60 % 2021-12-02 03:33:04,782 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.6 2021-12-02 03:33:04,796 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:33:04,799 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:33:04,799 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:09,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.60 % 2021-12-02 03:33:09,061 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:33:09,061 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:14,811 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:14,811 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.79 2021-12-02 03:33:14,834 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:33:14,838 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:33:14,838 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:19,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:19,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:33:19,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:24,851 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:24,851 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.79 2021-12-02 03:33:24,880 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:33:24,884 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:33:24,884 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:29,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:29,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:33:29,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:34,914 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:34,915 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.79 2021-12-02 03:33:34,933 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:33:34,936 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:33:34,936 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:39,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:39,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:33:39,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:44,950 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:44,950 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.79 2021-12-02 03:33:44,967 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:33:44,970 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:33:44,970 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:49,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:49,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:33:49,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:54,983 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:54,983 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.79 2021-12-02 03:33:55,002 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:33:55,005 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:33:55,005 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:33:59,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 03:33:59,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:33:59,142 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:05,016 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.99 % 2021-12-02 03:34:05,017 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.99 2021-12-02 03:34:05,033 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:34:05,036 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:34:05,036 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:09,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.99 % 2021-12-02 03:34:09,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:34:09,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:15,048 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.99 % 2021-12-02 03:34:15,048 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.99 2021-12-02 03:34:15,065 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:34:15,068 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:34:15,068 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:19,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.99 % 2021-12-02 03:34:19,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:34:19,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:25,079 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.99 % 2021-12-02 03:34:25,079 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 8.99 2021-12-02 03:34:25,095 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:34:25,096 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:34:25,096 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:29,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.18 % 2021-12-02 03:34:29,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:34:29,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:35,107 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.18 % 2021-12-02 03:34:35,107 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.18 2021-12-02 03:34:35,121 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:34:35,124 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:34:35,124 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:39,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:34:39,209 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:34:39,209 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:45,134 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:34:45,134 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.57 2021-12-02 03:34:45,148 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:34:45,150 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:34:45,150 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:49,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:34:49,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:34:49,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:55,160 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:34:55,160 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.57 2021-12-02 03:34:55,174 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:34:55,176 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:34:55,176 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:34:59,237 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:34:59,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:34:59,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:05,186 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:35:05,186 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.57 2021-12-02 03:35:05,198 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:35:05,200 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:35:05,200 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:09,255 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:35:09,260 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:35:09,260 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:15,210 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:35:15,211 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.57 2021-12-02 03:35:15,224 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:35:15,226 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:35:15,226 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:19,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:35:19,278 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:35:19,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:25,236 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 03:35:25,236 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.57 2021-12-02 03:35:25,253 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:35:25,256 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:35:25,256 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:29,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.77 % 2021-12-02 03:35:29,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:35:29,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:35,266 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.77 % 2021-12-02 03:35:35,267 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.77 2021-12-02 03:35:35,282 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:35:35,286 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:35:35,286 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:39,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.77 % 2021-12-02 03:35:39,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:35:39,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:45,296 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.77 % 2021-12-02 03:35:45,298 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.77 2021-12-02 03:35:45,313 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:35:45,315 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:35:45,315 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:49,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.77 % 2021-12-02 03:35:49,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:35:49,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:55,326 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.96 % 2021-12-02 03:35:55,326 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 9.96 2021-12-02 03:35:55,342 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:35:55,344 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:35:55,344 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:35:59,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.16 % 2021-12-02 03:35:59,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:35:59,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:05,357 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.16 % 2021-12-02 03:36:05,358 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.16 2021-12-02 03:36:05,377 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:36:05,378 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:36:05,378 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:09,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 03:36:09,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:36:09,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:15,391 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 03:36:15,391 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.35 2021-12-02 03:36:15,405 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:36:15,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:36:15,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:19,382 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 03:36:19,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:36:19,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:25,418 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 03:36:25,420 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.35 2021-12-02 03:36:25,434 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:36:25,436 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:36:25,436 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:29,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 03:36:29,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:36:29,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:35,447 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 03:36:35,447 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.35 2021-12-02 03:36:35,464 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:36:35,467 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:36:35,467 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:39,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:36:39,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:36:39,423 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:45,492 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:36:45,493 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.55 2021-12-02 03:36:45,513 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:36:45,515 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:36:45,516 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:49,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:36:49,438 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:36:49,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:55,528 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:36:55,529 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.55 2021-12-02 03:36:55,546 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:36:55,549 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:36:55,549 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:36:59,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:36:59,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:36:59,456 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:05,561 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:37:05,562 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.55 2021-12-02 03:37:05,584 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:37:05,585 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:37:05,586 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:09,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.55 % 2021-12-02 03:37:09,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:37:09,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:15,604 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 03:37:15,605 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.74 2021-12-02 03:37:15,618 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:37:15,621 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:37:15,621 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:19,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 03:37:19,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:37:19,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:25,632 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 03:37:25,634 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.74 2021-12-02 03:37:25,650 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:37:25,653 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:37:25,653 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:29,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 03:37:29,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:37:29,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:35,664 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 03:37:35,664 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 10.74 2021-12-02 03:37:35,683 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:37:35,686 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:37:35,686 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:39,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.94 % 2021-12-02 03:37:39,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:37:39,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:45,717 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 03:37:45,717 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.13 2021-12-02 03:37:45,733 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:37:45,736 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:37:45,736 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:49,545 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 03:37:49,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:37:49,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:55,749 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 03:37:55,750 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.13 2021-12-02 03:37:55,765 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:37:55,767 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:37:55,769 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:37:59,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 03:37:59,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:37:59,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:05,779 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 03:38:05,779 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.13 2021-12-02 03:38:05,796 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:38:05,798 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:38:05,799 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:09,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 03:38:09,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:38:09,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:15,809 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:15,809 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.53 2021-12-02 03:38:15,825 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:38:15,828 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:38:15,828 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:19,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:19,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:38:19,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:25,838 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:25,838 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.53 2021-12-02 03:38:25,854 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:38:25,857 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:38:25,857 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:29,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:29,630 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:38:29,630 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:35,868 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:35,868 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.53 2021-12-02 03:38:35,890 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:38:35,891 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:38:35,892 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:39,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:39,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:38:39,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:45,904 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:45,905 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.53 2021-12-02 03:38:45,924 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:38:45,927 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:38:45,928 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:49,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:49,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:38:49,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:55,941 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:55,941 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.53 2021-12-02 03:38:55,983 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:38:55,986 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:38:55,986 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:38:59,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:38:59,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:38:59,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:05,996 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:39:05,996 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.53 2021-12-02 03:39:06,015 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:39:06,017 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:39:06,019 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:09,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.53 % 2021-12-02 03:39:09,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:39:09,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:16,040 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.92 % 2021-12-02 03:39:16,042 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 11.92 2021-12-02 03:39:16,058 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:39:16,061 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:39:16,062 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:19,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.92 % 2021-12-02 03:39:19,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:39:19,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:26,073 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.11 % 2021-12-02 03:39:26,073 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.11 2021-12-02 03:39:26,091 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:39:26,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:39:26,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:29,742 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.11 % 2021-12-02 03:39:29,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:39:29,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:36,108 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:39:36,108 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.31 2021-12-02 03:39:36,153 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:39:36,155 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:39:36,155 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:39,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:39:39,770 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:39:39,770 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:46,167 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:39:46,167 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.31 2021-12-02 03:39:46,181 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:39:46,184 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:39:46,184 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:49,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:39:49,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:39:49,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:56,196 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:39:56,196 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.31 2021-12-02 03:39:56,213 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:39:56,214 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:39:56,214 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:39:59,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:39:59,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:39:59,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:06,226 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:40:06,226 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.31 2021-12-02 03:40:06,240 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:40:06,243 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:40:06,243 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:09,823 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.31 % 2021-12-02 03:40:09,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:40:09,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:16,252 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:16,253 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.5 2021-12-02 03:40:16,266 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:40:16,267 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:40:16,267 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:19,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:19,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:40:19,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:26,278 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:26,279 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.5 2021-12-02 03:40:26,292 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:40:26,293 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:40:26,295 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:29,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:29,869 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:40:29,869 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:36,305 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:36,305 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.5 2021-12-02 03:40:36,319 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:40:36,322 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:40:36,322 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:39,882 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:39,890 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:40:39,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:46,334 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:46,334 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.5 2021-12-02 03:40:46,351 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:40:46,352 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:40:46,354 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:49,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.50 % 2021-12-02 03:40:49,913 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:40:49,913 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:56,364 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.70 % 2021-12-02 03:40:56,364 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 12.7 2021-12-02 03:40:56,378 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:40:56,380 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:40:56,380 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:40:59,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.89 % 2021-12-02 03:40:59,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:40:59,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:06,390 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.09 % 2021-12-02 03:41:06,390 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.09 2021-12-02 03:41:06,404 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:41:06,405 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:41:06,405 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:09,941 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.09 % 2021-12-02 03:41:09,951 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:41:09,951 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:16,415 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.09 % 2021-12-02 03:41:16,417 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.09 2021-12-02 03:41:16,430 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:41:16,433 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:41:16,433 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:19,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.09 % 2021-12-02 03:41:19,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:41:19,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:26,444 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.09 % 2021-12-02 03:41:26,444 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.09 2021-12-02 03:41:26,463 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:41:26,466 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:41:26,466 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:29,989 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.09 % 2021-12-02 03:41:30,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:41:30,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:36,477 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:41:36,477 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 03:41:36,493 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:41:36,496 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:41:36,496 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:40,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:41:40,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:41:40,025 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:46,510 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:41:46,510 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 03:41:46,529 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:41:46,532 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:41:46,532 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:50,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:41:50,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:41:50,048 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:41:56,545 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:41:56,545 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 03:41:56,561 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:41:56,562 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:41:56,562 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:00,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:42:00,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:42:00,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:06,575 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:42:06,575 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 03:42:06,588 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:42:06,591 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:42:06,592 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:10,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:42:10,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:42:10,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:16,602 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 03:42:16,604 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 03:42:16,625 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:42:16,628 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:42:16,628 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:20,104 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.48 % 2021-12-02 03:42:20,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:42:20,112 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:26,641 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.48 % 2021-12-02 03:42:26,641 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.48 2021-12-02 03:42:26,661 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:42:26,664 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:42:26,665 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:30,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.48 % 2021-12-02 03:42:30,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:42:30,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:36,677 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.48 % 2021-12-02 03:42:36,677 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.48 2021-12-02 03:42:36,693 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:42:36,696 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:42:36,696 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:40,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.67 % 2021-12-02 03:42:40,155 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:42:40,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:46,707 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.67 % 2021-12-02 03:42:46,709 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.67 2021-12-02 03:42:46,726 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:42:46,730 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:42:46,730 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:50,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.67 % 2021-12-02 03:42:50,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:42:50,178 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:42:56,743 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.67 % 2021-12-02 03:42:56,743 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.67 2021-12-02 03:42:56,757 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:42:56,760 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:42:56,760 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:00,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.67 % 2021-12-02 03:43:00,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:43:00,201 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:06,773 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.87 % 2021-12-02 03:43:06,773 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 13.87 2021-12-02 03:43:06,792 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:43:06,796 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:43:06,796 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:10,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:10,224 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:43:10,224 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:16,808 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:16,808 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.06 2021-12-02 03:43:16,834 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:43:16,838 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:43:16,838 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:20,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:20,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:43:20,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:26,849 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:26,851 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.06 2021-12-02 03:43:26,865 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:43:26,868 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:43:26,868 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:30,255 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:30,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:43:30,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:36,878 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:36,880 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.06 2021-12-02 03:43:36,894 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:43:36,897 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:43:36,897 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:40,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 03:43:40,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:43:40,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:46,907 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.26 % 2021-12-02 03:43:46,907 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.26 2021-12-02 03:43:46,921 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:43:46,924 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:43:46,924 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:50,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.26 % 2021-12-02 03:43:50,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:43:50,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:43:56,934 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.26 % 2021-12-02 03:43:56,934 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.26 2021-12-02 03:43:56,950 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:43:56,953 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:43:56,953 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:00,308 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.26 % 2021-12-02 03:44:00,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:44:00,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:06,964 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.46 % 2021-12-02 03:44:06,964 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.46 2021-12-02 03:44:06,979 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:44:06,982 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:44:06,982 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:10,326 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.46 % 2021-12-02 03:44:10,334 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:44:10,334 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:16,993 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.46 % 2021-12-02 03:44:16,993 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.46 2021-12-02 03:44:17,012 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:44:17,015 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:44:17,015 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:20,346 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.46 % 2021-12-02 03:44:20,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:44:20,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:27,026 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.65 % 2021-12-02 03:44:27,026 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.65 2021-12-02 03:44:27,043 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:44:27,046 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:44:27,046 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:30,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.65 % 2021-12-02 03:44:30,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:44:30,374 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:37,062 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.65 % 2021-12-02 03:44:37,062 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.65 2021-12-02 03:44:37,078 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:44:37,079 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:44:37,079 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:40,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:44:40,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:44:40,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:47,091 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:44:47,091 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.85 2021-12-02 03:44:47,107 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:44:47,109 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:44:47,109 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:50,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:44:50,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:44:50,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:44:57,121 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:44:57,121 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.85 2021-12-02 03:44:57,154 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:44:57,158 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:44:57,158 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:00,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:45:00,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:45:00,451 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:07,170 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:45:07,170 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.85 2021-12-02 03:45:07,187 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:45:07,190 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:45:07,190 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:10,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:45:10,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:45:10,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:17,204 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:45:17,204 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 14.85 2021-12-02 03:45:17,221 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:45:17,223 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:45:17,224 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:20,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.85 % 2021-12-02 03:45:20,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:45:20,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:27,237 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.04 % 2021-12-02 03:45:27,237 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.04 2021-12-02 03:45:27,255 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:45:27,259 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:45:27,259 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:30,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:45:30,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:45:30,535 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:37,282 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:45:37,282 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.24 2021-12-02 03:45:37,303 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:45:37,309 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:45:37,309 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:40,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:45:40,561 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:45:40,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:47,322 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:45:47,322 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.24 2021-12-02 03:45:47,341 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:45:47,342 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:45:47,342 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:50,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:45:50,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:45:50,584 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:45:57,352 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:45:57,354 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.24 2021-12-02 03:45:57,371 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:45:57,372 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:45:57,372 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:00,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.24 % 2021-12-02 03:46:00,602 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:46:00,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:07,387 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.43 % 2021-12-02 03:46:07,388 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.43 2021-12-02 03:46:07,404 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:46:07,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:46:07,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:10,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:10,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:46:10,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:17,418 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:17,420 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.63 2021-12-02 03:46:17,437 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:46:17,440 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:46:17,440 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:20,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:20,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:46:20,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:27,453 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:27,453 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.63 2021-12-02 03:46:27,467 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:46:27,470 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:46:27,470 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:30,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:30,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:46:30,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:37,480 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:37,480 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.63 2021-12-02 03:46:37,497 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:46:37,500 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:46:37,500 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:40,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.63 % 2021-12-02 03:46:40,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:46:40,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:47,510 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.82 % 2021-12-02 03:46:47,510 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 15.82 2021-12-02 03:46:47,525 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:46:47,526 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:46:47,526 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:50,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.02 % 2021-12-02 03:46:50,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:46:50,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:46:57,538 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.02 % 2021-12-02 03:46:57,539 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.02 2021-12-02 03:46:57,559 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:46:57,561 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:46:57,562 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:00,714 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.02 % 2021-12-02 03:47:00,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:47:00,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:07,573 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.02 % 2021-12-02 03:47:07,573 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.02 2021-12-02 03:47:07,592 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:47:07,595 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:47:07,595 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:10,734 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.02 % 2021-12-02 03:47:10,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:47:10,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:17,608 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 03:47:17,608 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.21 2021-12-02 03:47:17,627 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:47:17,628 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:47:17,628 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:20,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 03:47:20,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:47:20,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:27,641 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 03:47:27,642 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.21 2021-12-02 03:47:27,664 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:47:27,667 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:47:27,667 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:30,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 03:47:30,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:47:30,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:37,680 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 03:47:37,680 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.21 2021-12-02 03:47:37,701 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:47:37,703 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:47:37,703 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:40,790 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 03:47:40,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:47:40,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:47,714 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:47:47,716 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.41 2021-12-02 03:47:47,750 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:47:47,753 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:47:47,753 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:50,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:47:50,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:47:50,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:47:57,765 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:47:57,765 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.41 2021-12-02 03:47:57,782 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:47:57,785 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:47:57,786 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:00,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:00,844 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:48:00,844 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:07,796 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:07,798 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.41 2021-12-02 03:48:07,815 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:48:07,819 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:48:07,819 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:10,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:10,862 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:48:10,862 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:17,832 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:17,832 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.41 2021-12-02 03:48:17,851 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:48:17,854 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:48:17,855 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:20,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:20,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:48:20,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:27,869 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:27,871 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 16.41 2021-12-02 03:48:27,887 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:48:27,890 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:48:27,890 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:30,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.41 % 2021-12-02 03:48:30,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:48:30,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:37,903 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:48:37,904 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.0 2021-12-02 03:48:37,921 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:48:37,923 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:48:37,924 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:40,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:48:40,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:48:40,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:47,938 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:48:47,938 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.0 2021-12-02 03:48:47,954 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:48:47,957 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:48:47,957 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:50,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:48:50,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:48:50,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:48:57,967 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:48:57,967 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.0 2021-12-02 03:48:57,982 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:48:57,983 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:48:57,983 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:00,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:49:00,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:49:00,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:07,993 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:49:07,994 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.0 2021-12-02 03:49:08,009 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:49:08,010 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:49:08,012 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:10,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:49:11,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:49:11,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:18,022 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.00 % 2021-12-02 03:49:18,022 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.0 2021-12-02 03:49:18,038 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:49:18,039 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:49:18,039 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:21,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.19 % 2021-12-02 03:49:21,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:49:21,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:28,052 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.19 % 2021-12-02 03:49:28,053 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.19 2021-12-02 03:49:28,082 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:49:28,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:49:28,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:31,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.19 % 2021-12-02 03:49:31,043 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:49:31,043 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:38,098 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.19 % 2021-12-02 03:49:38,098 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.19 2021-12-02 03:49:38,114 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:49:38,117 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:49:38,117 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:41,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.39 % 2021-12-02 03:49:41,063 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:49:41,063 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:48,128 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.39 % 2021-12-02 03:49:48,128 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.39 2021-12-02 03:49:48,145 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:49:48,148 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:49:48,148 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:51,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.39 % 2021-12-02 03:49:51,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:49:51,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:49:58,163 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.39 % 2021-12-02 03:49:58,163 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.39 2021-12-02 03:49:58,180 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:49:58,183 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:49:58,183 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:01,096 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.39 % 2021-12-02 03:50:01,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:50:01,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:08,194 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.58 % 2021-12-02 03:50:08,194 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.58 2021-12-02 03:50:08,213 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:50:08,216 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:50:08,217 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:11,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.58 % 2021-12-02 03:50:11,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:50:11,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:18,234 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.78 % 2021-12-02 03:50:18,236 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.78 2021-12-02 03:50:18,255 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:50:18,257 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:50:18,257 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:21,147 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.78 % 2021-12-02 03:50:21,153 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:50:21,153 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:28,273 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:28,273 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.97 2021-12-02 03:50:28,289 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:50:28,292 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:50:28,293 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:31,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:31,183 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:50:31,183 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:38,309 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:38,309 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.97 2021-12-02 03:50:38,325 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:50:38,328 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:50:38,328 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:41,201 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:41,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:50:41,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:48,346 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:48,346 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.97 2021-12-02 03:50:48,368 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:50:48,372 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:50:48,372 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:51,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:51,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:50:51,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:50:58,390 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:50:58,390 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.97 2021-12-02 03:50:58,408 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:50:58,413 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:50:58,413 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:01,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:51:01,265 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:51:01,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:08,431 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 03:51:08,431 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 17.97 2021-12-02 03:51:08,450 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:51:08,453 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:51:08,453 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:11,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:11,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:51:11,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:18,470 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:18,470 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.17 2021-12-02 03:51:18,484 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:51:18,486 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:51:18,486 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:21,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:21,331 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:51:21,331 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:28,502 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:28,503 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.17 2021-12-02 03:51:28,516 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:51:28,519 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:51:28,519 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:31,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:31,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:51:31,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:38,533 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:38,533 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.17 2021-12-02 03:51:38,552 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:51:38,555 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:51:38,555 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:41,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.17 % 2021-12-02 03:51:41,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:51:41,395 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:48,568 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.36 % 2021-12-02 03:51:48,569 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.36 2021-12-02 03:51:48,588 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:51:48,591 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:51:48,591 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:51,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:51:51,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:51:51,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:51:58,601 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:51:58,602 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.75 2021-12-02 03:51:58,615 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:51:58,618 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:51:58,618 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:01,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:01,443 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:52:01,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:08,630 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:08,631 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.75 2021-12-02 03:52:08,644 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:52:08,647 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:52:08,647 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:11,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:11,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:52:11,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:18,661 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:18,661 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.75 2021-12-02 03:52:18,677 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:52:18,678 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:52:18,678 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:21,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:21,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:52:21,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:28,693 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:28,693 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.75 2021-12-02 03:52:28,709 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:52:28,711 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:52:28,711 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:31,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.75 % 2021-12-02 03:52:31,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:52:31,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:38,726 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.95 % 2021-12-02 03:52:38,726 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.95 2021-12-02 03:52:38,742 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:52:38,744 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:52:38,744 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:41,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.95 % 2021-12-02 03:52:41,528 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:52:41,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:48,759 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.95 % 2021-12-02 03:52:48,759 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.95 2021-12-02 03:52:48,778 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:52:48,779 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:52:48,779 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:51,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.95 % 2021-12-02 03:52:51,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:52:51,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:52:58,792 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.95 % 2021-12-02 03:52:58,792 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 18.95 2021-12-02 03:52:58,809 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:52:58,812 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:52:58,812 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:01,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.95 % 2021-12-02 03:53:01,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:53:01,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:08,825 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 03:53:08,825 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.14 2021-12-02 03:53:08,839 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:53:08,842 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:53:08,842 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:11,586 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 03:53:11,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:53:11,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:18,859 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 03:53:18,861 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.14 2021-12-02 03:53:18,875 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:53:18,878 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:53:18,878 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:21,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 03:53:21,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:53:21,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:28,894 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 03:53:28,894 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.14 2021-12-02 03:53:28,910 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:53:28,913 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:53:28,913 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:31,624 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:53:31,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:53:31,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:38,927 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:53:38,927 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.54 2021-12-02 03:53:38,944 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:53:38,947 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:53:38,947 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:41,650 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:53:41,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:53:41,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:48,960 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:53:48,961 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.54 2021-12-02 03:53:48,976 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:53:48,979 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:53:48,980 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:51,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:53:51,678 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:53:51,678 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:53:58,992 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:53:58,993 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.54 2021-12-02 03:53:59,007 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:53:59,009 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:53:59,010 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:01,696 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.54 % 2021-12-02 03:54:01,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:54:01,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:09,023 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.73 % 2021-12-02 03:54:09,023 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.73 2021-12-02 03:54:09,038 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:54:09,040 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:54:09,040 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:11,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.73 % 2021-12-02 03:54:11,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:54:11,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:19,053 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.73 % 2021-12-02 03:54:19,055 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.73 2021-12-02 03:54:19,072 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:54:19,073 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:54:19,073 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:21,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.73 % 2021-12-02 03:54:21,757 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:54:21,757 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:29,085 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.73 % 2021-12-02 03:54:29,086 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.73 2021-12-02 03:54:29,101 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:54:29,104 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:54:29,104 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:31,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.73 % 2021-12-02 03:54:31,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:54:31,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:39,118 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.93 % 2021-12-02 03:54:39,118 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 19.93 2021-12-02 03:54:39,131 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:54:39,132 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:54:39,134 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:41,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.93 % 2021-12-02 03:54:41,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:54:41,806 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:49,147 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.12 % 2021-12-02 03:54:49,147 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 20.12 2021-12-02 03:54:49,161 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:54:49,164 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:54:49,164 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:51,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.12 % 2021-12-02 03:54:51,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 03:54:51,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:54:59,176 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.32 % 2021-12-02 03:54:59,176 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 20.32 2021-12-02 03:54:59,190 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:54:59,191 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:54:59,191 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:55:01,832 | test | INFO | MainThread | [rest_client:rebalance_reached:119] Rebalance reached >20.12% in 2224.80799985 seconds 2021-12-02 03:55:01,845 | test | DEBUG | MainThread | [rest_client:diag_eval:1453] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-02 03:55:01,845 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-02 03:55:01,848 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 03:55:01,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:55:02,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 03:55:02,237 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:55:02,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 03:55:02,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 03:55:02,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 03:55:02,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 03:55:02,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 03:55:02,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 03:55:02,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 03:55:02,371 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 03:55:02,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 03:55:02,428 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['204510 memcached\n'], []) 2021-12-02 03:55:02,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 03:55:02,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 03:55:02,520 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['204510 memcached\n'], []) 2021-12-02 03:55:02,522 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 03:55:02,526 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 03:55:02,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:55:02,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 03:55:02,868 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:55:02,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 03:55:02,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 03:55:02,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 03:55:02,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 03:55:02,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 03:55:02,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 03:55:02,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 03:55:02,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 03:55:02,967 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 03:55:03,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 03:55:03,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:03,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:03,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:55:03,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:55:03,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:03,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:03,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:55:03,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:55:03,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:03,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:03,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:55:03,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:55:03,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 03:55:03,371 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 03:55:03,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 03:55:03,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:03,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 03:55:03,411 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 03:55:03,411 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 03:55:03,415 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 03:55:03,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:55:03,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 03:55:03,835 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:55:03,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 03:55:03,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 03:55:03,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 03:55:03,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 03:55:03,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 03:55:03,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 03:55:03,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 03:55:03,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 03:55:03,990 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 03:55:04,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 03:55:04,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:04,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:04,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:55:04,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:55:04,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 03:55:04,163 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 03:55:04,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 03:55:04,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:04,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 03:55:04,219 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 03:55:04,220 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 03:55:04,223 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 03:55:04,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:55:04,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 03:55:04,678 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:55:04,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 03:55:04,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 03:55:04,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 03:55:04,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 03:55:04,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 03:55:04,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 03:55:04,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 03:55:04,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 03:55:04,838 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 03:55:04,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 03:55:04,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:04,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:04,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:55:05,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:55:05,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:05,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:05,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:55:05,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:55:05,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:05,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:05,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:55:05,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:55:05,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:05,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:05,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 03:55:05,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 03:55:05,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 03:55:05,450 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 03:55:05,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 03:55:05,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:05,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 03:55:05,497 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 03:55:05,497 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 03:55:05,500 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 03:55:05,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 03:55:05,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 03:55:05,957 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 03:55:05,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 03:55:05,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 03:55:05,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 03:55:06,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 03:55:06,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 03:55:06,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 03:55:06,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 03:55:06,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 03:55:06,095 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 03:55:06,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 03:55:06,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:06,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:06,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:55:06,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 03:55:06,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:06,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:06,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:55:06,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 03:55:06,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:06,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 03:55:06,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:55:06,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 03:55:06,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 03:55:06,595 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 03:55:06,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 03:55:06,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 03:55:06,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 03:55:06,650 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 03:55:06,651 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 03:55:06,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-02 03:55:06,657 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-02 03:55:09,206 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.32 % 2021-12-02 03:55:09,206 | test | INFO | pool-3-thread-20 | [task:check:384] Rebalance - status: running, progress: 20.32 2021-12-02 03:55:09,224 | infra | DEBUG | pool-3-thread-20 | [task:check:411] Wait before next rebalance progress check 2021-12-02 03:55:09,226 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 03:55:09,226 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 03:55:19,240 | test | ERROR | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1640] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'6a05b61f263522fee4953f95b66c23de', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=fdeb710d8fb30ab2661808c1a0f98c11', u'status': u'notRunning'} - rebalance failed 2021-12-02 03:55:19,269 | test | INFO | pool-3-thread-20 | [rest_client:print_UI_logs:2789] Latest logs from UI on 172.23.110.64: 2021-12-02 03:55:19,269 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638446114182L, u'shortText': u'message', u'serverTime': u'2021-12-02T03:55:14.182Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nW1201 20:33:57.140372 244731 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1201 21:53:53.328923 241683 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1201 23:20:14.538772 241640 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1202 01:27:17.196750 204658 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1202 02:48:05.196605 204661 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-02 03:55:19,269 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638446114175L, u'shortText': u'message', u'serverTime': u'2021-12-02T03:55:14.175Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.25989.110>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,stats_recv,4,\n [{file,"src/mc_client_binary.erl"},\n {line,163}]},\n {mc_client_binary,stats,4,\n [{file,"src/mc_client_binary.erl"},\n {line,455}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,475}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,224}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,322,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.31128.106>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.68\'],\n 322}},\n infinity]}}}}}.\nRebalance Operation Id = 4cc1039edf9bfabdf54520d56b8099ab'} 2021-12-02 03:55:19,270 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638446114141L, u'shortText': u'message', u'serverTime': u'2021-12-02T03:55:14.141Z', u'text': u'Worker <0.26600.110> (for action {move,\n {322,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.66\'],\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.68\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.25989.110>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n stats_recv,\n 4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 163}]},\n {mc_client_binary,\n stats,4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 455}]},\n {ns_memcached,\n do_handle_call,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 475}]},\n {ns_memcached,\n worker_loop,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 224}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 226}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 322,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.31128.106>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.68\'],\n 322}},\n infinity]}}}}'} 2021-12-02 03:55:19,272 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638446114130L, u'shortText': u'message', u'serverTime': u'2021-12-02T03:55:14.130Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-02 03:55:19,272 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638443795055L, u'shortText': u'message', u'serverTime': u'2021-12-02T03:16:35.055Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-02 03:55:19,272 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.68', u'tstamp': 1638443791868L, u'shortText': u'message', u'serverTime': u'2021-12-02T03:16:31.868Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.68\' in 0 seconds.'} 2021-12-02 03:55:19,273 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638443791555L, u'shortText': u'message', u'serverTime': u'2021-12-02T03:16:31.555Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-02 03:55:19,273 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.110.68', u'tstamp': 1638443791445L, u'shortText': u'message', u'serverTime': u'2021-12-02T03:16:31.445Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_storage_threads">>,\n <<"num_writer_threads">>,\n <<"scramsha_fallback_salt">>]'} 2021-12-02 03:55:19,273 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638443791315L, u'shortText': u'message', u'serverTime': u'2021-12-02T03:16:31.315Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.68',\n 'ns_1@172.23.110.66','ns_1@172.23.110.65'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 4cc1039edf9bfabdf54520d56b8099ab"} 2021-12-02 03:55:19,273 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2791] {u'code': 3, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.110.68', u'tstamp': 1638443791271L, u'shortText': u'message', u'serverTime': u'2021-12-02T03:16:31.271Z', u'text': u'Node ns_1@172.23.110.68 joined cluster'} 2021-12-02 03:55:19,275 | test | ERROR | pool-3-thread-20 | [task:call:296] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'6a05b61f263522fee4953f95b66c23de', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=fdeb710d8fb30ab2661808c1a0f98c11', u'status': u'notRunning'} - rebalance failed 2021-12-02 03:56:06,676 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-02 03:56:06,677 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 03:56:06,690 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-02 03:56:06,706 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.68:8091 2021-12-02 03:56:06,706 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 03:56:06,720 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.68:8091 is running 2021-12-02 03:56:06,730 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-02 03:56:06,730 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 03:56:06,740 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:06,749 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:06,749 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:08,759 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:08,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:08,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:10,778 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:10,785 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:10,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:12,798 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:12,803 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:12,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:14,819 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:14,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:14,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:16,844 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:16,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:16,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:18,865 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:18,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:18,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:20,884 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:20,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:20,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:22,903 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:22,908 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:22,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:24,921 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:24,927 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:24,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:26,941 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:26,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:26,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:28,963 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:28,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:28,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:30,980 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:30,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:30,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:33,003 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:33,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:33,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:35,023 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:35,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:35,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:37,045 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:37,051 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:37,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:39,065 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:39,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:39,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:41,088 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:41,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:41,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:43,111 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:43,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:43,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:45,132 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:45,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:45,140 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:47,151 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:47,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:47,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:49,174 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:49,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:49,181 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:51,193 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:51,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:51,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:53,210 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:53,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:53,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:55,230 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:55,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:55,237 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:57,255 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:57,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:57,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:56:59,275 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:56:59,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:56:59,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:01,292 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:01,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:01,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:03,312 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:03,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:03,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:05,331 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:05,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:05,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:07,352 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:07,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:07,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:09,371 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:09,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:09,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:11,388 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:11,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:11,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:13,411 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:13,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:13,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:15,433 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:15,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:15,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:17,451 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:17,459 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:17,459 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:19,470 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:19,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:19,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:21,489 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:21,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:21,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:23,509 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:23,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:23,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:25,532 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:25,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:25,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:27,551 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:27,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:27,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:29,572 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:29,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:29,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:31,591 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:31,596 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:31,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:33,611 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:33,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:33,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:35,630 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:35,637 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:35,637 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:37,648 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:37,655 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:37,655 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:39,667 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:39,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:39,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:41,687 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:41,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:41,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:43,707 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:43,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:43,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:45,727 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:45,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:45,734 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:47,750 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:47,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:47,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:49,769 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:49,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:49,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:51,789 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:51,798 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:51,798 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:53,811 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:53,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:53,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:55,832 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:55,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:55,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:57,852 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:57,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:57,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:57:59,871 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:57:59,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:57:59,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:01,892 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:01,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:01,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:03,913 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:03,920 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:03,920 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:05,937 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:05,943 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:05,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:07,957 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:07,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:07,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:09,977 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:09,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:09,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:12,000 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:12,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:12,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:14,026 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:14,033 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:14,035 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:16,048 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:16,053 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:16,053 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:18,068 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:18,073 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:18,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:20,088 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:20,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:20,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:22,108 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:22,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:22,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:24,131 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:24,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:24,140 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:26,157 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:26,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:26,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:28,178 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:28,186 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:28,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:30,198 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:30,204 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:30,206 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:32,219 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:32,227 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:32,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:39,232 | test | ERROR | MainThread | [rest_client:_http_request:829] Socket error while connecting to http://172.23.110.64:8091/nodes/self error timed out 2021-12-02 03:58:39,233 | test | ERROR | MainThread | [rest_client:is_ns_server_running:77] Server 172.23.110.64:8091 is unavailable 2021-12-02 03:58:39,239 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:39,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:41,253 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:41,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:41,265 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:43,279 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:43,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:43,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:45,299 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:45,308 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:45,308 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:47,323 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:47,331 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:47,331 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:49,345 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:49,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:49,352 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:51,367 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:51,372 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:51,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:53,387 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:53,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:53,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:55,407 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:55,415 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:55,415 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:57,433 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:57,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:57,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:58:59,453 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:58:59,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:58:59,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:01,473 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:01,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:01,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:03,493 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:03,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:03,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:05,512 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:05,517 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:05,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:07,546 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:07,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:07,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:09,572 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:09,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:09,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:11,591 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:11,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:11,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:13,611 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:13,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:13,619 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:15,637 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:15,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:15,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:17,657 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:17,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:17,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:19,680 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:19,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:19,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:21,701 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:21,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:21,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:23,720 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:23,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:23,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:25,743 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:25,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:25,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:27,766 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:27,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:27,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:29,789 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:29,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:29,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:31,811 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:31,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:31,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:33,832 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:33,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:33,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:35,855 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:35,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:35,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:37,877 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:37,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:37,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:39,898 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:39,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:39,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:41,918 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:41,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:41,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:43,937 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:43,946 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:43,946 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:45,957 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:45,964 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:45,964 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:47,980 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:47,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:47,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:50,002 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:50,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:50,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:52,026 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:52,033 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:52,033 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:54,048 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:54,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:54,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:56,069 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:56,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:56,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 03:59:58,089 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 03:59:58,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 03:59:58,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:00,114 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:00,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:00,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:02,137 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:02,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:02,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:04,158 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:04,165 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:04,165 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:06,178 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:06,186 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:06,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:08,200 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:08,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:08,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:10,220 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:10,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:10,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:12,240 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:12,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:12,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:14,263 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:14,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:14,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:16,290 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:16,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:16,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:18,312 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:18,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:18,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:20,332 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:20,339 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:20,339 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:22,354 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:22,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:22,362 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:24,377 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:24,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:24,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:26,398 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:26,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:26,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:28,418 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:28,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:28,426 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:30,440 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:30,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:30,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:32,460 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:32,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:32,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:34,482 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:34,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:34,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:36,500 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:36,507 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:36,507 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:38,522 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:38,528 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:38,528 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:40,546 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:40,553 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:40,553 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:42,568 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:42,573 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:42,573 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:44,585 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:44,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:44,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:46,605 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:46,612 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:46,612 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:48,628 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:48,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:48,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:50,650 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:50,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:50,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:52,670 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:52,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:52,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:54,690 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:54,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:54,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:56,710 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:56,719 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:56,719 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:00:58,734 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:00:58,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:00:58,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:00,756 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:00,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:00,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:02,779 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:02,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:02,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:04,801 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:04,808 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:04,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:06,821 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:06,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:06,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:08,842 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:08,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:08,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:10,859 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:10,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:10,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:12,881 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:12,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:12,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:14,901 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:14,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:14,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:16,921 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:16,928 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:16,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:18,947 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:18,954 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:18,956 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:20,967 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:20,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:20,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:22,990 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:22,997 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:22,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:25,012 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:25,017 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:25,017 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:27,032 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:27,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:27,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:29,052 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:29,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:29,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:31,072 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:31,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:31,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:33,092 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:33,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:33,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:35,117 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:35,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:35,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:37,138 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:37,145 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:37,145 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:39,157 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:39,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:39,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:41,178 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:41,186 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:41,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:43,200 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:43,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:43,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:45,220 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:45,229 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:45,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:47,244 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:47,252 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:47,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:49,269 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:49,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:49,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:51,289 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:51,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:51,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:53,311 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:53,318 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:53,318 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:55,329 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:55,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:55,336 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:57,349 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:57,357 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:57,357 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:01:59,371 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:01:59,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:01:59,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:01,395 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:01,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:01,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:03,436 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:03,441 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:03,441 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:05,456 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:05,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:05,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:07,476 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:07,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:07,484 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:09,496 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:09,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:09,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:11,516 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:11,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:11,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:13,538 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:13,545 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:13,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:15,559 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:15,566 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:15,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:17,578 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:17,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:17,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:19,598 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:19,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:19,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:21,621 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:21,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:21,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:23,644 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:23,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:23,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:25,664 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:25,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:25,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:27,681 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:27,688 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:27,688 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:29,704 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:29,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:29,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:31,727 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:31,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:31,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:33,747 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:33,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:33,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:35,767 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:35,775 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:35,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:37,789 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:37,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:37,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:39,809 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:39,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:39,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:41,829 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:41,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:41,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:43,852 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:43,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:43,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:45,872 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:45,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:45,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:47,894 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:47,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:47,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:49,915 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:49,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:49,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:51,934 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:51,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:51,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:53,956 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:53,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:53,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:55,980 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:55,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:55,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:02:57,999 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:02:58,005 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:02:58,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:00,019 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:00,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:00,028 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:02,039 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:02,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:02,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:04,059 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:04,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:04,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:06,078 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:06,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:06,086 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:08,101 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:08,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:08,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:10,121 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:10,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:10,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:12,138 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:12,145 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:12,145 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:14,157 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:14,163 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:14,163 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:16,178 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:16,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:16,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:18,197 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:18,204 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:18,204 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:20,216 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:20,221 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:20,221 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:22,240 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:22,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:22,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:24,262 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:24,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:24,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:26,282 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:26,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:26,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:28,303 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:28,311 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:28,311 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:30,326 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:30,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:30,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:32,349 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:32,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:32,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:34,375 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:34,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:34,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:36,398 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:36,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:36,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:38,421 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:38,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:38,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:40,443 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:40,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:40,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:42,464 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:42,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:42,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:44,487 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:44,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:44,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:46,507 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:46,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:46,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:48,532 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:48,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:48,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:50,553 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:50,561 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:50,561 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:52,575 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:52,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:52,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:54,598 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:54,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:54,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:56,622 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:56,630 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:56,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:03:58,645 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 04:03:58,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 04:03:58,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 04:04:00,665 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-02 04:04:00,677 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-02 04:04:00,677 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 04:04:00,687 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-02 04:04:00,688 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 04:04:00,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 04:04:00,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 04:04:01,012 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 04:04:01,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 04:04:01,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 04:04:01,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 04:04:01,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 04:04:01,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 04:04:01,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 04:04:01,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 04:04:01,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 04:04:01,132 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 04:04:01,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 04:04:01,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:01,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 04:04:01,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 04:04:01,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 04:04:01,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:01,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 04:04:01,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 04:04:01,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 04:04:01,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:01,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 04:04:01,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 04:04:01,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 04:04:01,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 04:04:01,463 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 04:04:01,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 04:04:01,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:01,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 04:04:01,496 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 04:04:01,496 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 04:04:01,500 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 04:04:01,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 04:04:01,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 04:04:01,901 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 04:04:01,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 04:04:01,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 04:04:01,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 04:04:01,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 04:04:01,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 04:04:01,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 04:04:02,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 04:04:02,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 04:04:02,042 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 04:04:02,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 04:04:02,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:02,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 04:04:02,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 04:04:02,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 04:04:02,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 04:04:02,154 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 04:04:02,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 04:04:02,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:02,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 04:04:02,187 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 04:04:02,187 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 04:04:02,190 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 04:04:02,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 04:04:02,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 04:04:02,589 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 04:04:02,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 04:04:02,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 04:04:02,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 04:04:02,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 04:04:02,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 04:04:02,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 04:04:02,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 04:04:02,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 04:04:02,710 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 04:04:02,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 04:04:02,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:02,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 04:04:02,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 04:04:02,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 04:04:02,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:03,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 04:04:03,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 04:04:03,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 04:04:03,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:03,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 04:04:03,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 04:04:03,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 04:04:03,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:03,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 04:04:03,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 04:04:03,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 04:04:03,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:03,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 04:04:03,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 04:04:03,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 04:04:03,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 04:04:03,411 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 04:04:03,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 04:04:03,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:03,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 04:04:03,461 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 04:04:03,463 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 04:04:03,466 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 04:04:03,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 04:04:03,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 04:04:03,868 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 04:04:03,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 04:04:03,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 04:04:03,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 04:04:03,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 04:04:03,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 04:04:03,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 04:04:03,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 04:04:03,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 04:04:03,993 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 04:04:04,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 04:04:04,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:04,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 04:04:04,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 04:04:04,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 04:04:04,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:04,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 04:04:04,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 04:04:04,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 04:04:04,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:04,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 04:04:04,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 04:04:04,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 04:04:04,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 04:04:04,322 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 04:04:04,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 04:04:04,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 04:04:04,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 04:04:04,358 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 04:04:04,358 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 04:04:04,358 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.110.68]_OUT=[]_1638443786.97 2021-12-02 04:04:04,359 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 20 2021-12-02 04:04:04,388 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1638446644.36 2021-12-02 04:04:04,388 | infra | INFO | pool-3-thread-21 | [task:start_task:89] Thread 'Rebalance_task_IN=[]_OUT=[]_1638446644.36' started 2021-12-02 04:04:04,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-02 04:04:04,395 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-02 04:04:04,415 | test | DEBUG | pool-3-thread-21 | [rest_client:rebalance:1431] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-02 04:04:04,473 | test | DEBUG | pool-3-thread-21 | [rest_client:rebalance:1436] Rebalance operation started 2021-12-02 04:04:04,476 | test | INFO | pool-3-thread-21 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1817-enterprise | 5.0653322213 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1817-enterprise | 2.38326848249 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1817-enterprise | 2.21189694508 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1817-enterprise | 1.58571190119 | Cluster node | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-02 04:04:04,489 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 04:04:04,489 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 04:04:04,506 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:04:04,507 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:04:04,509 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:04:14,523 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 04:04:14,523 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 04:04:14,539 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:04:14,542 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:04:14,542 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:04:24,566 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 04:04:24,568 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 04:04:24,584 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:04:24,586 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:04:24,586 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:04:34,599 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 04:04:34,601 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 04:04:34,615 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:04:34,617 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:04:34,618 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:04:44,630 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 04:04:44,630 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 04:04:44,644 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:04:44,648 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:04:44,648 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:04:54,661 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 04:04:54,661 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 04:04:54,676 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:04:54,677 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:04:54,677 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:05:04,690 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 04:05:04,690 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 04:05:04,703 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:05:04,706 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:05:04,707 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:05:14,719 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.24 % 2021-12-02 04:05:14,719 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.24 2021-12-02 04:05:14,736 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:05:14,739 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:05:14,739 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:05:24,752 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.49 % 2021-12-02 04:05:24,752 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.49 2021-12-02 04:05:24,766 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:05:24,769 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:05:24,769 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:05:34,782 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.49 % 2021-12-02 04:05:34,782 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.49 2021-12-02 04:05:34,799 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:05:34,802 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:05:34,802 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:05:44,816 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.49 % 2021-12-02 04:05:44,818 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.49 2021-12-02 04:05:44,844 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:05:44,845 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:05:44,845 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:05:54,859 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.99 % 2021-12-02 04:05:54,861 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.99 2021-12-02 04:05:54,881 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:05:54,882 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:05:54,884 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:06:04,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.99 % 2021-12-02 04:06:04,407 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 0.99 2021-12-02 04:06:04,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.99 % 2021-12-02 04:06:04,423 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:06:04,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:06:04,898 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.99 % 2021-12-02 04:06:04,898 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 0.99 2021-12-02 04:06:04,914 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:06:04,917 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:06:04,917 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:06:14,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.23 % 2021-12-02 04:06:14,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:06:14,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:06:14,930 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.23 % 2021-12-02 04:06:14,930 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 1.23 2021-12-02 04:06:14,948 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:06:14,953 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:06:14,953 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:06:24,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.23 % 2021-12-02 04:06:24,464 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:06:24,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:06:24,966 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.23 % 2021-12-02 04:06:24,966 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 1.23 2021-12-02 04:06:24,989 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:06:24,992 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:06:24,992 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:06:34,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.23 % 2021-12-02 04:06:34,484 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:06:34,484 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:06:35,007 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.23 % 2021-12-02 04:06:35,009 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 1.23 2021-12-02 04:06:35,029 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:06:35,033 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:06:35,035 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:06:44,497 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.47 % 2021-12-02 04:06:44,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:06:44,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:06:45,048 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.47 % 2021-12-02 04:06:45,049 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 1.47 2021-12-02 04:06:45,065 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:06:45,069 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:06:45,069 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:06:54,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.47 % 2021-12-02 04:06:54,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:06:54,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:06:55,081 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.47 % 2021-12-02 04:06:55,081 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 1.47 2021-12-02 04:06:55,099 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:06:55,105 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:06:55,105 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:07:04,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.47 % 2021-12-02 04:07:04,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:07:04,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:07:05,118 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.47 % 2021-12-02 04:07:05,118 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 1.47 2021-12-02 04:07:05,138 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:07:05,140 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:07:05,141 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:07:14,558 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.72 % 2021-12-02 04:07:14,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:07:14,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:07:15,154 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.72 % 2021-12-02 04:07:15,154 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 1.72 2021-12-02 04:07:15,171 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:07:15,174 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:07:15,174 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:07:24,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.98 % 2021-12-02 04:07:24,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:07:24,586 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:07:25,187 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.98 % 2021-12-02 04:07:25,188 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 1.98 2021-12-02 04:07:25,207 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:07:25,210 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:07:25,210 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:07:34,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.98 % 2021-12-02 04:07:34,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:07:34,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:07:35,233 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.98 % 2021-12-02 04:07:35,233 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 1.98 2021-12-02 04:07:35,256 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:07:35,259 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:07:35,259 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:07:44,624 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.98 % 2021-12-02 04:07:44,632 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:07:44,632 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:07:45,275 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.98 % 2021-12-02 04:07:45,275 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 1.98 2021-12-02 04:07:45,292 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:07:45,296 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:07:45,296 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:07:54,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.22 % 2021-12-02 04:07:54,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:07:54,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:07:55,312 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.22 % 2021-12-02 04:07:55,312 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 2.22 2021-12-02 04:07:55,329 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:07:55,332 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:07:55,332 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:08:04,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.22 % 2021-12-02 04:08:04,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:08:04,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:08:05,345 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.22 % 2021-12-02 04:08:05,345 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 2.22 2021-12-02 04:08:05,359 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:08:05,361 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:08:05,361 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:08:14,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.46 % 2021-12-02 04:08:14,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:08:14,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:08:15,371 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.46 % 2021-12-02 04:08:15,371 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 2.46 2021-12-02 04:08:15,385 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:08:15,388 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:08:15,388 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:08:24,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.46 % 2021-12-02 04:08:24,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:08:24,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:08:25,401 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.46 % 2021-12-02 04:08:25,401 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 2.46 2021-12-02 04:08:25,417 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:08:25,420 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:08:25,420 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:08:34,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.96 % 2021-12-02 04:08:34,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:08:34,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:08:35,430 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.96 % 2021-12-02 04:08:35,431 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 2.96 2021-12-02 04:08:35,446 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:08:35,447 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:08:35,447 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:08:44,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.96 % 2021-12-02 04:08:44,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:08:44,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:08:45,460 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.96 % 2021-12-02 04:08:45,460 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 2.96 2021-12-02 04:08:45,476 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:08:45,477 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:08:45,477 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:08:54,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.96 % 2021-12-02 04:08:54,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:08:54,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:08:55,490 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.96 % 2021-12-02 04:08:55,490 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 2.96 2021-12-02 04:08:55,506 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:08:55,509 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:08:55,509 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:09:04,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.96 % 2021-12-02 04:09:04,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:09:04,783 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:09:05,523 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.96 % 2021-12-02 04:09:05,523 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 2.96 2021-12-02 04:09:05,540 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:09:05,543 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:09:05,543 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:09:14,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.96 % 2021-12-02 04:09:14,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:09:14,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:09:15,559 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.96 % 2021-12-02 04:09:15,561 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 2.96 2021-12-02 04:09:15,578 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:09:15,579 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:09:15,581 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:09:24,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.96 % 2021-12-02 04:09:24,821 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:09:24,821 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:09:25,595 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.96 % 2021-12-02 04:09:25,595 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 2.96 2021-12-02 04:09:25,611 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:09:25,614 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:09:25,614 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:09:34,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.44 % 2021-12-02 04:09:34,839 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:09:34,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:09:35,625 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.44 % 2021-12-02 04:09:35,625 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 3.44 2021-12-02 04:09:35,641 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:09:35,642 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:09:35,642 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:09:44,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.44 % 2021-12-02 04:09:44,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:09:44,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:09:45,654 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.44 % 2021-12-02 04:09:45,655 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 3.44 2021-12-02 04:09:45,670 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:09:45,671 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:09:45,671 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:09:54,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:09:54,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:09:54,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:09:55,684 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:09:55,684 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 3.94 2021-12-02 04:09:55,698 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:09:55,700 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:09:55,700 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:10:04,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:10:04,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:10:04,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:10:05,711 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:10:05,711 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 3.94 2021-12-02 04:10:05,726 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:10:05,729 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:10:05,729 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:10:14,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:10:14,918 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:10:14,920 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:10:15,742 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:10:15,742 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 3.94 2021-12-02 04:10:15,756 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:10:15,759 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:10:15,759 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:10:24,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:10:24,938 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:10:24,938 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:10:25,769 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:10:25,769 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 3.94 2021-12-02 04:10:25,783 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:10:25,786 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:10:25,786 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:10:34,951 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:10:34,959 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:10:34,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:10:35,799 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:10:35,799 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 3.94 2021-12-02 04:10:35,812 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:10:35,815 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:10:35,815 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:10:44,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:10:44,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:10:44,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:10:45,825 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:10:45,825 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 3.94 2021-12-02 04:10:45,841 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:10:45,842 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:10:45,842 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:10:55,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:10:55,015 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:10:55,015 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:10:55,855 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.94 % 2021-12-02 04:10:55,857 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 3.94 2021-12-02 04:10:55,877 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:10:55,878 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:10:55,878 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:11:05,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.43 % 2021-12-02 04:11:05,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:11:05,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:11:05,891 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.43 % 2021-12-02 04:11:05,891 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 4.43 2021-12-02 04:11:05,905 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:11:05,907 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:11:05,907 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:11:15,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.43 % 2021-12-02 04:11:15,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:11:15,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:11:15,918 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.43 % 2021-12-02 04:11:15,920 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 4.43 2021-12-02 04:11:15,940 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:11:15,943 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:11:15,943 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:11:25,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:11:25,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:11:25,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:11:25,957 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:11:25,957 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 4.93 2021-12-02 04:11:25,973 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:11:25,976 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:11:25,976 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:11:35,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:11:35,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:11:35,112 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:11:35,992 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:11:35,992 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 4.93 2021-12-02 04:11:36,010 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:11:36,013 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:11:36,013 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:11:45,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:11:45,132 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:11:45,132 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:11:46,026 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:11:46,026 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 4.93 2021-12-02 04:11:46,042 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:11:46,046 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:11:46,046 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:11:55,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:11:55,155 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:11:55,155 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:11:56,058 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:11:56,058 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 4.93 2021-12-02 04:11:56,075 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:11:56,078 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:11:56,078 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:12:05,168 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:12:05,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:12:05,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:12:06,092 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:12:06,092 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 4.93 2021-12-02 04:12:06,108 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:12:06,111 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:12:06,111 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:12:15,191 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:12:15,201 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:12:15,201 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:12:16,125 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:12:16,125 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 4.93 2021-12-02 04:12:16,141 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:12:16,142 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:12:16,144 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:12:25,214 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:12:25,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:12:25,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:12:26,155 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.93 % 2021-12-02 04:12:26,155 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 4.93 2021-12-02 04:12:26,171 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:12:26,174 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:12:26,174 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:12:35,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.43 % 2021-12-02 04:12:35,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:12:35,242 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:12:36,190 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.43 % 2021-12-02 04:12:36,190 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 5.43 2021-12-02 04:12:36,206 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:12:36,209 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:12:36,210 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:12:45,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.66 % 2021-12-02 04:12:45,262 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:12:45,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:12:46,221 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.66 % 2021-12-02 04:12:46,221 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 5.66 2021-12-02 04:12:46,236 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:12:46,240 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:12:46,240 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:12:55,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:12:55,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:12:55,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:12:56,252 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:12:56,252 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 5.92 2021-12-02 04:12:56,267 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:12:56,270 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:12:56,270 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:13:05,293 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:13:05,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:13:05,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:13:06,283 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:13:06,283 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 5.92 2021-12-02 04:13:06,299 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:13:06,302 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:13:06,302 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:13:15,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:13:15,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:13:15,323 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:13:16,315 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:13:16,315 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 5.92 2021-12-02 04:13:16,332 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:13:16,335 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:13:16,335 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:13:25,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:13:25,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:13:25,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:13:26,345 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:13:26,346 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 5.92 2021-12-02 04:13:26,364 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:13:26,367 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:13:26,367 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:13:35,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:13:35,369 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:13:35,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:13:36,378 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:13:36,380 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 5.92 2021-12-02 04:13:36,395 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:13:36,398 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:13:36,398 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:13:45,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:13:45,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:13:45,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:13:46,411 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:13:46,413 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 5.92 2021-12-02 04:13:46,427 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:13:46,428 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:13:46,430 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:13:55,400 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:13:55,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:13:55,408 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:13:56,440 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.92 % 2021-12-02 04:13:56,440 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 5.92 2021-12-02 04:13:56,459 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:13:56,460 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:13:56,460 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:14:05,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.15 % 2021-12-02 04:14:05,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:14:05,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:14:06,471 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.15 % 2021-12-02 04:14:06,471 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 6.15 2021-12-02 04:14:06,486 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:14:06,487 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:14:06,487 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:14:15,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.15 % 2021-12-02 04:14:15,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:14:15,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:14:16,499 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.15 % 2021-12-02 04:14:16,500 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 6.15 2021-12-02 04:14:16,520 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:14:16,523 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:14:16,523 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:14:25,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.90 % 2021-12-02 04:14:25,466 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:14:25,466 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:14:26,535 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.90 % 2021-12-02 04:14:26,535 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 6.9 2021-12-02 04:14:26,549 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:14:26,552 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:14:26,552 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:14:35,477 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.90 % 2021-12-02 04:14:35,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:14:35,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:14:36,565 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.90 % 2021-12-02 04:14:36,565 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 6.9 2021-12-02 04:14:36,585 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:14:36,588 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:14:36,588 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:14:45,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.90 % 2021-12-02 04:14:45,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:14:45,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:14:46,602 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.90 % 2021-12-02 04:14:46,602 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 6.9 2021-12-02 04:14:46,621 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:14:46,625 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:14:46,625 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:14:55,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.90 % 2021-12-02 04:14:55,520 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:14:55,520 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:14:56,641 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.90 % 2021-12-02 04:14:56,641 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 6.9 2021-12-02 04:14:56,661 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:14:56,664 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:14:56,664 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:15:05,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.90 % 2021-12-02 04:15:05,538 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:15:05,538 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:15:06,681 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.90 % 2021-12-02 04:15:06,683 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 6.9 2021-12-02 04:15:06,701 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:15:06,704 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:15:06,706 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:15:15,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.90 % 2021-12-02 04:15:15,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:15:15,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:15:16,720 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.90 % 2021-12-02 04:15:16,720 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 6.9 2021-12-02 04:15:16,737 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:15:16,740 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:15:16,740 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:15:25,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 04:15:25,573 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:15:25,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:15:26,753 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 04:15:26,753 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 04:15:26,769 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:15:26,770 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:15:26,770 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:15:35,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 04:15:35,596 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:15:35,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:15:36,782 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.39 % 2021-12-02 04:15:36,782 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 7.39 2021-12-02 04:15:36,796 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:15:36,798 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:15:36,798 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:15:45,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.63 % 2021-12-02 04:15:45,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:15:45,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:15:46,811 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.63 % 2021-12-02 04:15:46,811 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 7.63 2021-12-02 04:15:46,825 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:15:46,828 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:15:46,828 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:15:55,637 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:15:55,645 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:15:55,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:15:56,841 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:15:56,842 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 7.87 2021-12-02 04:15:56,857 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:15:56,858 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:15:56,858 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:16:05,658 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:16:05,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:16:05,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:16:06,875 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:16:06,875 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 7.87 2021-12-02 04:16:06,891 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:16:06,892 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:16:06,894 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:16:15,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:16:15,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:16:15,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:16:16,905 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:16:16,907 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 7.87 2021-12-02 04:16:16,924 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:16:16,926 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:16:16,926 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:16:25,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:16:25,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:16:25,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:16:26,940 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:16:26,940 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 7.87 2021-12-02 04:16:26,956 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:16:26,957 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:16:26,957 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:16:35,729 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:16:35,737 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:16:35,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:16:36,970 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:16:36,971 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 7.87 2021-12-02 04:16:36,986 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:16:36,987 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:16:36,987 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:16:45,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:16:45,762 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:16:45,762 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:16:47,000 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:16:47,000 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 7.87 2021-12-02 04:16:47,020 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:16:47,023 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:16:47,023 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:16:55,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:16:55,780 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:16:55,780 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:16:57,039 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.87 % 2021-12-02 04:16:57,039 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 7.87 2021-12-02 04:16:57,061 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:16:57,063 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:16:57,063 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:17:05,793 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.36 % 2021-12-02 04:17:05,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:17:05,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:17:07,075 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.36 % 2021-12-02 04:17:07,075 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 8.36 2021-12-02 04:17:07,094 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:17:07,096 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:17:07,096 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:17:15,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.36 % 2021-12-02 04:17:15,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:17:15,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:17:17,109 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.36 % 2021-12-02 04:17:17,109 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 8.36 2021-12-02 04:17:17,125 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:17:17,127 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:17:17,127 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:17:25,831 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.86 % 2021-12-02 04:17:25,836 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:17:25,836 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:17:27,140 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.86 % 2021-12-02 04:17:27,140 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 8.86 2021-12-02 04:17:27,155 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:17:27,157 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:17:27,158 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:17:35,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.86 % 2021-12-02 04:17:35,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:17:35,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:17:37,171 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.86 % 2021-12-02 04:17:37,173 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 8.86 2021-12-02 04:17:37,191 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:17:37,194 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:17:37,194 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:17:45,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.86 % 2021-12-02 04:17:45,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:17:45,877 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:17:47,207 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.86 % 2021-12-02 04:17:47,207 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 8.86 2021-12-02 04:17:47,223 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:17:47,226 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:17:47,226 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:17:55,890 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.86 % 2021-12-02 04:17:55,895 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:17:55,895 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:17:57,240 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.86 % 2021-12-02 04:17:57,240 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 8.86 2021-12-02 04:17:57,255 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:17:57,257 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:17:57,257 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:18:05,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.86 % 2021-12-02 04:18:05,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:18:05,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:18:07,269 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.86 % 2021-12-02 04:18:07,269 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 8.86 2021-12-02 04:18:07,285 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:18:07,286 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:18:07,288 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:18:15,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.10 % 2021-12-02 04:18:15,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:18:15,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:18:17,299 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.10 % 2021-12-02 04:18:17,299 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 9.1 2021-12-02 04:18:17,319 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:18:17,322 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:18:17,322 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:18:25,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.34 % 2021-12-02 04:18:25,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:18:25,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:18:27,335 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.34 % 2021-12-02 04:18:27,335 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 9.34 2021-12-02 04:18:27,349 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:18:27,352 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:18:27,352 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:18:35,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.34 % 2021-12-02 04:18:35,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:18:35,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:18:37,365 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.34 % 2021-12-02 04:18:37,365 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 9.34 2021-12-02 04:18:37,381 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:18:37,384 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:18:37,384 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:18:45,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.34 % 2021-12-02 04:18:45,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:18:45,990 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:18:47,397 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.60 % 2021-12-02 04:18:47,397 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 9.6 2021-12-02 04:18:47,411 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:18:47,414 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:18:47,414 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:18:56,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.60 % 2021-12-02 04:18:56,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:18:56,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:18:57,427 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.60 % 2021-12-02 04:18:57,427 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 9.6 2021-12-02 04:18:57,441 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:18:57,444 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:18:57,444 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:19:06,020 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.85 % 2021-12-02 04:19:06,028 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:19:06,028 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:19:07,456 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.85 % 2021-12-02 04:19:07,457 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 9.85 2021-12-02 04:19:07,470 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:19:07,471 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:19:07,473 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:19:16,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.85 % 2021-12-02 04:19:16,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:19:16,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:19:17,483 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.85 % 2021-12-02 04:19:17,484 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 9.85 2021-12-02 04:19:17,499 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:19:17,500 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:19:17,500 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:19:26,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.85 % 2021-12-02 04:19:26,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:19:26,086 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:19:27,513 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.85 % 2021-12-02 04:19:27,513 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 9.85 2021-12-02 04:19:27,530 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:19:27,532 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:19:27,532 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:19:36,102 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.85 % 2021-12-02 04:19:36,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:19:36,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:19:37,546 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.85 % 2021-12-02 04:19:37,546 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 9.85 2021-12-02 04:19:37,561 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:19:37,562 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:19:37,562 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:19:46,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.85 % 2021-12-02 04:19:46,132 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:19:46,132 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:19:47,578 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.85 % 2021-12-02 04:19:47,578 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 9.85 2021-12-02 04:19:47,594 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:19:47,596 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:19:47,596 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:19:56,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.09 % 2021-12-02 04:19:56,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:19:56,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:19:57,608 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.32 % 2021-12-02 04:19:57,608 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 10.32 2021-12-02 04:19:57,624 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:19:57,625 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:19:57,625 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:20:06,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.32 % 2021-12-02 04:20:06,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:20:06,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:20:07,638 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.32 % 2021-12-02 04:20:07,638 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 10.32 2021-12-02 04:20:07,654 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:20:07,657 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:20:07,657 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:20:16,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.58 % 2021-12-02 04:20:16,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:20:16,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:20:17,671 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.58 % 2021-12-02 04:20:17,671 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 10.58 2021-12-02 04:20:17,686 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:20:17,688 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:20:17,688 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:20:26,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.82 % 2021-12-02 04:20:26,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:20:26,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:20:27,703 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.82 % 2021-12-02 04:20:27,703 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 10.82 2021-12-02 04:20:27,719 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:20:27,721 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:20:27,721 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:20:36,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.82 % 2021-12-02 04:20:36,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:20:36,249 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:20:37,733 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.82 % 2021-12-02 04:20:37,733 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 10.82 2021-12-02 04:20:37,760 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:20:37,763 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:20:37,763 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:20:46,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.82 % 2021-12-02 04:20:46,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:20:46,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:20:47,775 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.82 % 2021-12-02 04:20:47,776 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 10.82 2021-12-02 04:20:47,790 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:20:47,792 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:20:47,792 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:20:56,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.82 % 2021-12-02 04:20:56,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:20:56,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:20:57,805 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.82 % 2021-12-02 04:20:57,805 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 10.82 2021-12-02 04:20:57,819 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:20:57,822 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:20:57,822 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:21:06,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.06 % 2021-12-02 04:21:06,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:21:06,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:21:07,836 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.06 % 2021-12-02 04:21:07,836 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 11.06 2021-12-02 04:21:07,851 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:21:07,854 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:21:07,854 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:21:16,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.06 % 2021-12-02 04:21:16,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:21:16,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:21:17,865 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.31 % 2021-12-02 04:21:17,867 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 11.31 2021-12-02 04:21:17,881 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:21:17,882 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:21:17,882 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:21:26,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.31 % 2021-12-02 04:21:26,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:21:26,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:21:27,898 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.31 % 2021-12-02 04:21:27,898 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 11.31 2021-12-02 04:21:27,914 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:21:27,915 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:21:27,915 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:21:36,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.31 % 2021-12-02 04:21:36,397 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:21:36,397 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:21:37,928 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.31 % 2021-12-02 04:21:37,928 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 11.31 2021-12-02 04:21:37,944 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:21:37,946 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:21:37,946 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:21:46,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.56 % 2021-12-02 04:21:46,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:21:46,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:21:47,959 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.56 % 2021-12-02 04:21:47,959 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 11.56 2021-12-02 04:21:47,973 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:21:47,974 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:21:47,974 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:21:56,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.81 % 2021-12-02 04:21:56,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:21:56,441 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:21:57,986 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.81 % 2021-12-02 04:21:57,987 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 11.81 2021-12-02 04:21:58,002 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:21:58,003 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:21:58,003 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:22:06,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.81 % 2021-12-02 04:22:06,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:22:06,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:22:08,016 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.81 % 2021-12-02 04:22:08,016 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 11.81 2021-12-02 04:22:08,030 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:22:08,032 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:22:08,032 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:22:16,474 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.81 % 2021-12-02 04:22:16,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:22:16,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:22:18,046 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.81 % 2021-12-02 04:22:18,046 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 11.81 2021-12-02 04:22:18,061 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:22:18,062 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:22:18,062 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:22:26,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.81 % 2021-12-02 04:22:26,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:22:26,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:22:28,075 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.81 % 2021-12-02 04:22:28,075 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 11.81 2021-12-02 04:22:28,092 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:22:28,095 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:22:28,095 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:22:36,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.06 % 2021-12-02 04:22:36,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:22:36,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:22:38,108 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.29 % 2021-12-02 04:22:38,108 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 12.29 2021-12-02 04:22:38,127 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:22:38,130 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:22:38,130 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:22:46,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.29 % 2021-12-02 04:22:46,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:22:46,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:22:48,144 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.29 % 2021-12-02 04:22:48,144 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 12.29 2021-12-02 04:22:48,161 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:22:48,163 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:22:48,164 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:22:56,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.29 % 2021-12-02 04:22:56,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:22:56,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:22:58,177 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.29 % 2021-12-02 04:22:58,177 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 12.29 2021-12-02 04:22:58,194 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:22:58,198 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:22:58,198 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:23:06,561 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.55 % 2021-12-02 04:23:06,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:23:06,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:23:08,211 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.55 % 2021-12-02 04:23:08,213 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 12.55 2021-12-02 04:23:08,230 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:23:08,233 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:23:08,233 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:23:16,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.55 % 2021-12-02 04:23:16,584 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:23:16,584 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:23:18,249 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.55 % 2021-12-02 04:23:18,250 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 12.55 2021-12-02 04:23:18,266 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:23:18,270 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:23:18,270 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:23:26,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.55 % 2021-12-02 04:23:26,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:23:26,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:23:28,282 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.55 % 2021-12-02 04:23:28,282 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 12.55 2021-12-02 04:23:28,299 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:23:28,302 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:23:28,302 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:23:36,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.55 % 2021-12-02 04:23:36,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:23:36,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:23:38,313 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.55 % 2021-12-02 04:23:38,315 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 12.55 2021-12-02 04:23:38,328 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:23:38,332 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:23:38,332 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:23:46,630 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.04 % 2021-12-02 04:23:46,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:23:46,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:23:48,344 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.04 % 2021-12-02 04:23:48,344 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 13.04 2021-12-02 04:23:48,358 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:23:48,361 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:23:48,362 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:23:56,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.04 % 2021-12-02 04:23:56,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:23:56,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:23:58,378 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.04 % 2021-12-02 04:23:58,378 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 13.04 2021-12-02 04:23:58,397 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:23:58,400 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:23:58,400 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:24:06,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 04:24:06,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:24:06,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:24:08,413 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 04:24:08,414 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 04:24:08,427 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:24:08,430 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:24:08,430 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:24:16,688 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 04:24:16,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:24:16,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:24:18,444 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 04:24:18,444 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 04:24:18,459 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:24:18,461 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:24:18,461 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:24:26,709 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.53 % 2021-12-02 04:24:26,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:24:26,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:24:28,473 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.53 % 2021-12-02 04:24:28,473 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 13.53 2021-12-02 04:24:28,490 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:24:28,492 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:24:28,493 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:24:36,729 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.53 % 2021-12-02 04:24:36,737 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:24:36,737 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:24:38,503 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.53 % 2021-12-02 04:24:38,503 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 13.53 2021-12-02 04:24:38,516 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:24:38,519 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:24:38,519 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:24:46,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.53 % 2021-12-02 04:24:46,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:24:46,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:24:48,529 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.53 % 2021-12-02 04:24:48,529 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 13.53 2021-12-02 04:24:48,545 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:24:48,546 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:24:48,546 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:24:56,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.53 % 2021-12-02 04:24:56,783 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:24:56,783 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:24:58,559 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.53 % 2021-12-02 04:24:58,559 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 13.53 2021-12-02 04:24:58,575 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:24:58,578 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:24:58,578 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:25:06,795 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.78 % 2021-12-02 04:25:06,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:25:06,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:25:08,591 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.78 % 2021-12-02 04:25:08,591 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 13.78 2021-12-02 04:25:08,612 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:25:08,615 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:25:08,615 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:25:16,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.78 % 2021-12-02 04:25:16,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:25:16,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:25:18,628 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.78 % 2021-12-02 04:25:18,628 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 13.78 2021-12-02 04:25:18,641 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:25:18,642 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:25:18,644 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:25:26,829 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.03 % 2021-12-02 04:25:26,835 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:25:26,835 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:25:28,655 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.03 % 2021-12-02 04:25:28,655 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 14.03 2021-12-02 04:25:28,670 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:25:28,673 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:25:28,673 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:25:36,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.28 % 2021-12-02 04:25:36,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:25:36,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:25:38,686 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.28 % 2021-12-02 04:25:38,686 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 14.28 2021-12-02 04:25:38,704 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:25:38,709 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:25:38,709 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:25:46,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.52 % 2021-12-02 04:25:46,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:25:46,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:25:48,721 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.52 % 2021-12-02 04:25:48,721 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 14.52 2021-12-02 04:25:48,737 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:25:48,740 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:25:48,740 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:25:56,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.52 % 2021-12-02 04:25:56,890 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:25:56,890 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:25:58,752 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.52 % 2021-12-02 04:25:58,752 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 14.52 2021-12-02 04:25:58,769 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:25:58,772 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:25:58,772 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:26:06,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.52 % 2021-12-02 04:26:06,910 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:26:06,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:26:08,795 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.52 % 2021-12-02 04:26:08,795 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 14.52 2021-12-02 04:26:08,812 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:26:08,813 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:26:08,815 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:26:16,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.52 % 2021-12-02 04:26:16,933 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:26:16,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:26:18,825 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.52 % 2021-12-02 04:26:18,826 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 14.52 2021-12-02 04:26:18,839 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:26:18,842 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:26:18,842 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:26:26,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.76 % 2021-12-02 04:26:26,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:26:26,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:26:28,854 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.76 % 2021-12-02 04:26:28,854 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 14.76 2021-12-02 04:26:28,871 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:26:28,872 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:26:28,872 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:26:36,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.76 % 2021-12-02 04:26:36,971 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:26:36,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:26:38,887 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.76 % 2021-12-02 04:26:38,887 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 14.76 2021-12-02 04:26:38,910 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:26:38,914 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:26:38,914 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:26:46,992 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.01 % 2021-12-02 04:26:46,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:26:46,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:26:48,928 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.01 % 2021-12-02 04:26:48,928 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 15.01 2021-12-02 04:26:48,947 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:26:48,950 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:26:48,950 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:26:57,010 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.26 % 2021-12-02 04:26:57,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:26:57,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:26:58,967 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.26 % 2021-12-02 04:26:58,967 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 15.26 2021-12-02 04:26:58,986 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:26:58,990 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:26:58,990 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:27:07,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.26 % 2021-12-02 04:27:07,033 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:27:07,033 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:27:09,006 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.26 % 2021-12-02 04:27:09,006 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 15.26 2021-12-02 04:27:09,026 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:27:09,028 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:27:09,028 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:27:17,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.26 % 2021-12-02 04:27:17,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:27:17,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:27:19,046 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.26 % 2021-12-02 04:27:19,048 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 15.26 2021-12-02 04:27:19,062 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:27:19,065 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:27:19,065 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:27:27,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.50 % 2021-12-02 04:27:27,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:27:27,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:27:29,079 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.50 % 2021-12-02 04:27:29,081 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 15.5 2021-12-02 04:27:29,095 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:27:29,098 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:27:29,098 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:27:37,089 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.50 % 2021-12-02 04:27:37,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:27:37,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:27:39,115 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.50 % 2021-12-02 04:27:39,115 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 15.5 2021-12-02 04:27:39,132 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:27:39,134 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:27:39,135 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:27:47,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.50 % 2021-12-02 04:27:47,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:27:47,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:27:49,151 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.75 % 2021-12-02 04:27:49,151 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 15.75 2021-12-02 04:27:49,167 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:27:49,170 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:27:49,170 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:27:57,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.75 % 2021-12-02 04:27:57,150 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:27:57,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:27:59,183 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.75 % 2021-12-02 04:27:59,183 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 15.75 2021-12-02 04:27:59,200 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:27:59,201 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:27:59,201 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:28:07,168 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.75 % 2021-12-02 04:28:07,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:28:07,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:28:09,217 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.75 % 2021-12-02 04:28:09,217 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 15.75 2021-12-02 04:28:09,233 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:28:09,236 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:28:09,236 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:28:17,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.24 % 2021-12-02 04:28:17,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:28:17,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:28:19,260 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.24 % 2021-12-02 04:28:19,260 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 16.24 2021-12-02 04:28:19,276 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:28:19,279 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:28:19,279 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:28:27,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.24 % 2021-12-02 04:28:27,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:28:27,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:28:29,292 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.24 % 2021-12-02 04:28:29,292 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 16.24 2021-12-02 04:28:29,319 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:28:29,322 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:28:29,322 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:28:37,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.24 % 2021-12-02 04:28:37,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:28:37,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:28:39,335 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.24 % 2021-12-02 04:28:39,335 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 16.24 2021-12-02 04:28:39,351 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:28:39,354 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:28:39,354 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:28:47,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.24 % 2021-12-02 04:28:47,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:28:47,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:28:49,367 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.24 % 2021-12-02 04:28:49,367 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 16.24 2021-12-02 04:28:49,382 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:28:49,385 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:28:49,385 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:28:57,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.48 % 2021-12-02 04:28:57,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:28:57,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:28:59,398 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.48 % 2021-12-02 04:28:59,398 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 16.48 2021-12-02 04:28:59,414 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:28:59,417 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:28:59,417 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:29:07,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.48 % 2021-12-02 04:29:07,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:29:07,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:29:09,447 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.48 % 2021-12-02 04:29:09,447 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 16.48 2021-12-02 04:29:09,466 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:29:09,469 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:29:09,469 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:29:17,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.48 % 2021-12-02 04:29:17,354 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:29:17,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:29:19,483 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.48 % 2021-12-02 04:29:19,483 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 16.48 2021-12-02 04:29:19,500 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:29:19,502 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:29:19,503 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:29:27,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.72 % 2021-12-02 04:29:27,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:29:27,374 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:29:29,515 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.72 % 2021-12-02 04:29:29,516 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 16.72 2021-12-02 04:29:29,530 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:29:29,532 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:29:29,532 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:29:37,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.97 % 2021-12-02 04:29:37,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:29:37,395 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:29:39,545 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.22 % 2021-12-02 04:29:39,545 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 17.22 2021-12-02 04:29:39,559 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:29:39,561 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:29:39,561 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:29:47,410 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.22 % 2021-12-02 04:29:47,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:29:47,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:29:49,572 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.22 % 2021-12-02 04:29:49,572 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 17.22 2021-12-02 04:29:49,586 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:29:49,588 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:29:49,588 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:29:57,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.22 % 2021-12-02 04:29:57,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:29:57,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:29:59,601 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.22 % 2021-12-02 04:29:59,601 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 17.22 2021-12-02 04:29:59,615 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:29:59,618 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:29:59,618 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:30:07,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.22 % 2021-12-02 04:30:07,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:30:07,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:30:09,630 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.22 % 2021-12-02 04:30:09,630 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 17.22 2021-12-02 04:30:09,644 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:30:09,645 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:30:09,645 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:30:17,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.22 % 2021-12-02 04:30:17,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:30:17,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:30:19,657 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.47 % 2021-12-02 04:30:19,657 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 17.47 2021-12-02 04:30:19,671 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:30:19,674 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:30:19,674 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:30:27,497 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.47 % 2021-12-02 04:30:27,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:30:27,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:30:29,687 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.47 % 2021-12-02 04:30:29,687 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 17.47 2021-12-02 04:30:29,703 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:30:29,706 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:30:29,706 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:30:37,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.47 % 2021-12-02 04:30:37,530 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:30:37,530 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:30:39,719 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.47 % 2021-12-02 04:30:39,719 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 17.47 2021-12-02 04:30:39,736 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:30:39,737 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:30:39,739 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:30:47,543 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.47 % 2021-12-02 04:30:47,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:30:47,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:30:49,753 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.47 % 2021-12-02 04:30:49,753 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 17.47 2021-12-02 04:30:49,770 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:30:49,773 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:30:49,775 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:30:57,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.47 % 2021-12-02 04:30:57,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:30:57,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:30:59,788 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.47 % 2021-12-02 04:30:59,788 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 17.47 2021-12-02 04:30:59,805 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:30:59,809 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:30:59,809 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:31:07,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.71 % 2021-12-02 04:31:07,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:31:07,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:31:09,821 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.71 % 2021-12-02 04:31:09,822 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 17.71 2021-12-02 04:31:09,835 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:31:09,836 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:31:09,838 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:31:17,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.20 % 2021-12-02 04:31:17,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:31:17,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:31:19,851 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.20 % 2021-12-02 04:31:19,851 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 18.2 2021-12-02 04:31:19,868 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:31:19,871 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:31:19,871 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:31:27,624 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.20 % 2021-12-02 04:31:27,630 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:31:27,630 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:31:29,882 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.20 % 2021-12-02 04:31:29,882 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 18.2 2021-12-02 04:31:29,897 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:31:29,898 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:31:29,898 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:31:37,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.20 % 2021-12-02 04:31:37,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:31:37,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:31:39,910 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.46 % 2021-12-02 04:31:39,911 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 18.46 2021-12-02 04:31:39,926 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:31:39,927 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:31:39,928 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:31:47,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.46 % 2021-12-02 04:31:47,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:31:47,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:31:49,940 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.46 % 2021-12-02 04:31:49,940 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 18.46 2021-12-02 04:31:49,954 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:31:49,957 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:31:49,957 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:31:57,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.46 % 2021-12-02 04:31:57,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:31:57,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:31:59,967 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.46 % 2021-12-02 04:31:59,969 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 18.46 2021-12-02 04:31:59,982 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:31:59,983 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:31:59,983 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:32:07,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.46 % 2021-12-02 04:32:07,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:32:07,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:32:09,996 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.46 % 2021-12-02 04:32:09,997 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 18.46 2021-12-02 04:32:10,013 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:32:10,015 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:32:10,015 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:32:17,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.70 % 2021-12-02 04:32:17,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:32:17,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:32:20,029 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.70 % 2021-12-02 04:32:20,030 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 18.7 2021-12-02 04:32:20,046 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:32:20,048 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:32:20,048 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:32:27,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.70 % 2021-12-02 04:32:27,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:32:27,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:32:30,061 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.70 % 2021-12-02 04:32:30,062 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 18.7 2021-12-02 04:32:30,082 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:32:30,084 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:32:30,084 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:32:37,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.70 % 2021-12-02 04:32:37,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:32:37,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:32:40,096 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.70 % 2021-12-02 04:32:40,096 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 18.7 2021-12-02 04:32:40,111 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:32:40,114 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:32:40,114 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:32:47,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.94 % 2021-12-02 04:32:47,783 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:32:47,785 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:32:50,128 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.18 % 2021-12-02 04:32:50,128 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 19.18 2021-12-02 04:32:50,148 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:32:50,151 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:32:50,151 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:32:57,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.18 % 2021-12-02 04:32:57,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:32:57,806 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:33:00,164 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.18 % 2021-12-02 04:33:00,164 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 19.18 2021-12-02 04:33:00,183 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:33:00,186 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:33:00,186 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:33:07,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.44 % 2021-12-02 04:33:07,826 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:33:07,826 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:33:10,197 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.44 % 2021-12-02 04:33:10,198 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 19.44 2021-12-02 04:33:10,213 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:33:10,216 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:33:10,216 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:33:17,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.44 % 2021-12-02 04:33:17,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:33:17,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:33:20,229 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.44 % 2021-12-02 04:33:20,230 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 19.44 2021-12-02 04:33:20,246 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:33:20,249 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:33:20,250 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:33:27,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.44 % 2021-12-02 04:33:27,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:33:27,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:33:30,260 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.44 % 2021-12-02 04:33:30,262 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 19.44 2021-12-02 04:33:30,275 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:33:30,276 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:33:30,276 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:33:37,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.44 % 2021-12-02 04:33:37,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:33:37,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:33:40,289 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.44 % 2021-12-02 04:33:40,289 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 19.44 2021-12-02 04:33:40,305 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:33:40,309 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:33:40,309 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:33:47,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.69 % 2021-12-02 04:33:47,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:33:47,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:33:50,322 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.69 % 2021-12-02 04:33:50,323 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 19.69 2021-12-02 04:33:50,339 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:33:50,344 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:33:50,344 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:33:57,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.69 % 2021-12-02 04:33:57,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:33:57,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:34:00,361 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.69 % 2021-12-02 04:34:00,361 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 19.69 2021-12-02 04:34:00,377 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:34:00,380 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:34:00,380 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:34:07,926 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.69 % 2021-12-02 04:34:07,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:34:07,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:34:10,392 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.69 % 2021-12-02 04:34:10,392 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 19.69 2021-12-02 04:34:10,408 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:34:10,411 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:34:10,411 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:34:17,943 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.94 % 2021-12-02 04:34:17,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:34:17,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:34:20,426 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.18 % 2021-12-02 04:34:20,426 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 20.18 2021-12-02 04:34:20,440 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:34:20,444 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:34:20,444 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:34:27,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.42 % 2021-12-02 04:34:27,966 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:34:27,966 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:34:30,454 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.42 % 2021-12-02 04:34:30,456 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 20.42 2021-12-02 04:34:30,470 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:34:30,473 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:34:30,473 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:34:37,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.42 % 2021-12-02 04:34:37,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:34:37,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:34:40,483 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.42 % 2021-12-02 04:34:40,484 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 20.42 2021-12-02 04:34:40,499 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:34:40,502 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:34:40,502 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:34:47,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.42 % 2021-12-02 04:34:48,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:34:48,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:34:50,515 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.42 % 2021-12-02 04:34:50,516 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 20.42 2021-12-02 04:34:50,532 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:34:50,535 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:34:50,536 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:34:58,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.42 % 2021-12-02 04:34:58,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:34:58,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:35:00,549 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.42 % 2021-12-02 04:35:00,549 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 20.42 2021-12-02 04:35:00,565 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:35:00,568 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:35:00,568 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:35:08,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.42 % 2021-12-02 04:35:08,038 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:35:08,038 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:35:10,578 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.42 % 2021-12-02 04:35:10,579 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 20.42 2021-12-02 04:35:10,594 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:35:10,596 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:35:10,596 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:35:18,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.42 % 2021-12-02 04:35:18,056 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:35:18,056 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:35:20,608 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.42 % 2021-12-02 04:35:20,608 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 20.42 2021-12-02 04:35:20,622 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:35:20,625 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:35:20,625 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:35:28,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.67 % 2021-12-02 04:35:28,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:35:28,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:35:30,637 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.67 % 2021-12-02 04:35:30,637 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 20.67 2021-12-02 04:35:30,651 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:35:30,654 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:35:30,654 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:35:38,086 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.67 % 2021-12-02 04:35:38,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:35:38,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:35:40,665 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.67 % 2021-12-02 04:35:40,665 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 20.67 2021-12-02 04:35:40,678 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:35:40,680 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:35:40,680 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:35:48,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.92 % 2021-12-02 04:35:48,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:35:48,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:35:50,693 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.16 % 2021-12-02 04:35:50,693 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 21.16 2021-12-02 04:35:50,709 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:35:50,711 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:35:50,711 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:35:58,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.41 % 2021-12-02 04:35:58,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:35:58,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:36:00,723 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.41 % 2021-12-02 04:36:00,723 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 21.41 2021-12-02 04:36:00,739 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:36:00,740 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:36:00,742 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:36:08,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.41 % 2021-12-02 04:36:08,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:36:08,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:36:10,756 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.41 % 2021-12-02 04:36:10,756 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 21.41 2021-12-02 04:36:10,772 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:36:10,773 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:36:10,775 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:36:18,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.41 % 2021-12-02 04:36:18,176 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:36:18,176 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:36:20,789 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.41 % 2021-12-02 04:36:20,789 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 21.41 2021-12-02 04:36:20,806 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:36:20,809 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:36:20,809 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:36:28,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.41 % 2021-12-02 04:36:28,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:36:28,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:36:30,823 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.41 % 2021-12-02 04:36:30,823 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 21.41 2021-12-02 04:36:30,841 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:36:30,845 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:36:30,845 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:36:38,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.41 % 2021-12-02 04:36:38,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:36:38,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:36:40,857 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.41 % 2021-12-02 04:36:40,858 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 21.41 2021-12-02 04:36:40,875 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:36:40,878 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:36:40,878 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:36:48,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.41 % 2021-12-02 04:36:48,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:36:48,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:36:50,890 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.65 % 2021-12-02 04:36:50,890 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 21.65 2021-12-02 04:36:50,904 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:36:50,907 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:36:50,907 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:36:58,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.65 % 2021-12-02 04:36:58,275 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:36:58,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:37:00,918 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.65 % 2021-12-02 04:37:00,918 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 21.65 2021-12-02 04:37:00,934 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:37:00,936 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:37:00,936 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:37:08,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.65 % 2021-12-02 04:37:08,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:37:08,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:37:10,947 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.65 % 2021-12-02 04:37:10,947 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 21.65 2021-12-02 04:37:10,963 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:37:10,966 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:37:10,966 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:37:18,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.16 % 2021-12-02 04:37:18,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:37:18,323 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:37:20,977 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.16 % 2021-12-02 04:37:20,977 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 22.16 2021-12-02 04:37:20,993 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:37:20,996 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:37:20,996 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:37:28,341 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.40 % 2021-12-02 04:37:28,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:37:28,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:37:31,009 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.40 % 2021-12-02 04:37:31,009 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 22.4 2021-12-02 04:37:31,023 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:37:31,026 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:37:31,026 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:37:38,364 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.40 % 2021-12-02 04:37:38,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:37:38,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:37:41,039 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.40 % 2021-12-02 04:37:41,039 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 22.4 2021-12-02 04:37:41,053 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:37:41,056 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:37:41,056 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:37:48,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.40 % 2021-12-02 04:37:48,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:37:48,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:37:51,069 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.40 % 2021-12-02 04:37:51,071 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 22.4 2021-12-02 04:37:51,088 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:37:51,089 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:37:51,089 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:37:58,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.64 % 2021-12-02 04:37:58,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:37:58,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:38:01,104 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.64 % 2021-12-02 04:38:01,104 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 22.64 2021-12-02 04:38:01,119 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:38:01,121 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:38:01,122 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:38:08,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.64 % 2021-12-02 04:38:08,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:38:08,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:38:11,135 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.64 % 2021-12-02 04:38:11,135 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 22.64 2021-12-02 04:38:11,150 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:38:11,153 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:38:11,153 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:38:18,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.64 % 2021-12-02 04:38:18,466 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:38:18,466 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:38:21,170 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.64 % 2021-12-02 04:38:21,170 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 22.64 2021-12-02 04:38:21,186 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:38:21,187 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:38:21,187 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:38:28,479 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.64 % 2021-12-02 04:38:28,487 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:38:28,487 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:38:31,203 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.64 % 2021-12-02 04:38:31,203 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 22.64 2021-12-02 04:38:31,224 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:38:31,229 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:38:31,230 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:38:38,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.64 % 2021-12-02 04:38:38,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:38:38,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:38:41,244 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.64 % 2021-12-02 04:38:41,244 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 22.64 2021-12-02 04:38:41,263 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:38:41,267 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:38:41,269 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:38:48,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.88 % 2021-12-02 04:38:48,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:38:48,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:38:51,282 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.88 % 2021-12-02 04:38:51,282 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 22.88 2021-12-02 04:38:51,313 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:38:51,315 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:38:51,315 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:38:58,543 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.88 % 2021-12-02 04:38:58,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:38:58,553 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:39:01,328 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.38 % 2021-12-02 04:39:01,328 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 23.38 2021-12-02 04:39:01,344 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:39:01,346 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:39:01,346 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:39:08,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.38 % 2021-12-02 04:39:08,573 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:39:08,573 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:39:11,358 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.38 % 2021-12-02 04:39:11,358 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 23.38 2021-12-02 04:39:11,374 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:39:11,377 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:39:11,377 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:39:18,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.38 % 2021-12-02 04:39:18,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:39:18,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:39:21,388 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.38 % 2021-12-02 04:39:21,388 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 23.38 2021-12-02 04:39:21,404 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:39:21,407 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:39:21,407 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:39:28,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.38 % 2021-12-02 04:39:28,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:39:28,619 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:39:31,421 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.38 % 2021-12-02 04:39:31,421 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 23.38 2021-12-02 04:39:31,437 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:39:31,440 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:39:31,440 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:39:38,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.63 % 2021-12-02 04:39:38,640 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:39:38,640 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:39:41,453 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.63 % 2021-12-02 04:39:41,453 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 23.63 2021-12-02 04:39:41,467 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:39:41,470 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:39:41,470 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:39:48,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.63 % 2021-12-02 04:39:48,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:39:48,665 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:39:51,484 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.63 % 2021-12-02 04:39:51,484 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 23.63 2021-12-02 04:39:51,500 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:39:51,505 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:39:51,505 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:39:58,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.87 % 2021-12-02 04:39:58,688 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:39:58,688 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:40:01,520 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.87 % 2021-12-02 04:40:01,520 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 23.87 2021-12-02 04:40:01,536 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:40:01,539 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:40:01,539 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:40:08,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.87 % 2021-12-02 04:40:08,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:40:08,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:40:11,552 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.87 % 2021-12-02 04:40:11,552 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 23.87 2021-12-02 04:40:11,569 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:40:11,572 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:40:11,572 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:40:18,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.87 % 2021-12-02 04:40:18,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:40:18,734 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:40:21,585 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.87 % 2021-12-02 04:40:21,585 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 23.87 2021-12-02 04:40:21,622 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:40:21,625 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:40:21,625 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:40:28,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.11 % 2021-12-02 04:40:28,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:40:28,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:40:31,641 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.11 % 2021-12-02 04:40:31,642 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 24.11 2021-12-02 04:40:31,661 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:40:31,664 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:40:31,665 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:40:38,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.36 % 2021-12-02 04:40:38,772 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:40:38,772 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:40:41,678 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.36 % 2021-12-02 04:40:41,678 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 24.36 2021-12-02 04:40:41,694 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:40:41,696 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:40:41,697 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:40:48,783 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.61 % 2021-12-02 04:40:48,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:40:48,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:40:51,710 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.61 % 2021-12-02 04:40:51,710 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 24.61 2021-12-02 04:40:51,724 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:40:51,726 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:40:51,726 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:40:58,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.61 % 2021-12-02 04:40:58,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:40:58,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:41:01,739 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.61 % 2021-12-02 04:41:01,740 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 24.61 2021-12-02 04:41:01,753 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:41:01,756 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:41:01,756 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:41:08,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.61 % 2021-12-02 04:41:08,835 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:41:08,835 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:41:11,769 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.61 % 2021-12-02 04:41:11,769 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 24.61 2021-12-02 04:41:11,789 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:41:11,792 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:41:11,792 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:41:18,849 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.61 % 2021-12-02 04:41:18,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:41:18,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:41:21,816 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.61 % 2021-12-02 04:41:21,818 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 24.61 2021-12-02 04:41:21,835 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:41:21,838 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:41:21,838 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:41:28,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.61 % 2021-12-02 04:41:28,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:41:28,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:41:31,851 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.85 % 2021-12-02 04:41:31,851 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 24.85 2021-12-02 04:41:31,868 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:41:31,871 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:41:31,871 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:41:38,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.85 % 2021-12-02 04:41:38,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:41:38,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:41:41,888 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.85 % 2021-12-02 04:41:41,888 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 24.85 2021-12-02 04:41:41,907 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:41:41,911 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:41:41,911 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:41:48,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.85 % 2021-12-02 04:41:48,920 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:41:48,920 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:41:51,924 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.85 % 2021-12-02 04:41:51,924 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 24.85 2021-12-02 04:41:51,940 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:41:51,943 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:41:51,943 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:41:58,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.85 % 2021-12-02 04:41:58,938 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:41:58,938 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:42:01,953 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.85 % 2021-12-02 04:42:01,954 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 24.85 2021-12-02 04:42:01,969 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:42:01,970 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:42:01,970 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:42:08,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.35 % 2021-12-02 04:42:08,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:42:08,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:42:11,982 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.59 % 2021-12-02 04:42:11,982 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 25.59 2021-12-02 04:42:11,996 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:42:11,999 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:42:11,999 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:42:18,974 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.59 % 2021-12-02 04:42:18,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:42:18,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:42:22,010 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.59 % 2021-12-02 04:42:22,010 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 25.59 2021-12-02 04:42:22,025 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:42:22,026 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:42:22,026 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:42:28,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.59 % 2021-12-02 04:42:28,999 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:42:29,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:42:32,039 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.59 % 2021-12-02 04:42:32,039 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 25.59 2021-12-02 04:42:32,056 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:42:32,059 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:42:32,059 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:42:39,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.59 % 2021-12-02 04:42:39,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:42:39,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:42:42,072 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.59 % 2021-12-02 04:42:42,072 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 25.59 2021-12-02 04:42:42,092 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:42:42,095 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:42:42,095 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:42:49,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.59 % 2021-12-02 04:42:49,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:42:49,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:42:52,107 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.59 % 2021-12-02 04:42:52,108 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 25.59 2021-12-02 04:42:52,124 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:42:52,125 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:42:52,125 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:42:59,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.59 % 2021-12-02 04:42:59,056 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:42:59,056 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:43:02,138 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.59 % 2021-12-02 04:43:02,138 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 25.59 2021-12-02 04:43:02,160 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:43:02,161 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:43:02,161 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:43:09,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.83 % 2021-12-02 04:43:09,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:43:09,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:43:12,174 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.83 % 2021-12-02 04:43:12,174 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 25.83 2021-12-02 04:43:12,193 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:43:12,196 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:43:12,196 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:43:19,091 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.83 % 2021-12-02 04:43:19,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:43:19,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:43:22,209 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.83 % 2021-12-02 04:43:22,209 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 25.83 2021-12-02 04:43:22,230 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:43:22,233 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:43:22,233 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:43:29,109 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.83 % 2021-12-02 04:43:29,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:43:29,117 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:43:32,246 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.83 % 2021-12-02 04:43:32,246 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 25.83 2021-12-02 04:43:32,262 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:43:32,265 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:43:32,265 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:43:39,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.83 % 2021-12-02 04:43:39,135 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:43:39,135 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:43:42,282 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.83 % 2021-12-02 04:43:42,282 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 25.83 2021-12-02 04:43:42,313 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:43:42,315 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:43:42,315 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:43:49,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.33 % 2021-12-02 04:43:49,155 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:43:49,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:43:52,332 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.58 % 2021-12-02 04:43:52,332 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 26.58 2021-12-02 04:43:52,348 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:43:52,349 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:43:52,349 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:43:59,168 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.58 % 2021-12-02 04:43:59,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:43:59,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:44:02,361 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.58 % 2021-12-02 04:44:02,362 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 26.58 2021-12-02 04:44:02,378 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:44:02,380 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:44:02,380 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:44:09,186 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.58 % 2021-12-02 04:44:09,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:44:09,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:44:12,394 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.58 % 2021-12-02 04:44:12,394 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 26.58 2021-12-02 04:44:12,407 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:44:12,411 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:44:12,411 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:44:19,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.58 % 2021-12-02 04:44:19,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:44:19,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:44:22,424 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.58 % 2021-12-02 04:44:22,424 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 26.58 2021-12-02 04:44:22,441 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:44:22,444 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:44:22,446 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:44:29,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.58 % 2021-12-02 04:44:29,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:44:29,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:44:32,457 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.58 % 2021-12-02 04:44:32,457 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 26.58 2021-12-02 04:44:32,471 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:44:32,474 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:44:32,476 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:44:39,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.82 % 2021-12-02 04:44:39,255 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:44:39,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:44:42,487 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.82 % 2021-12-02 04:44:42,489 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 26.82 2021-12-02 04:44:42,503 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:44:42,506 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:44:42,506 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:44:49,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.82 % 2021-12-02 04:44:49,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:44:49,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:44:52,519 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.82 % 2021-12-02 04:44:52,519 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 26.82 2021-12-02 04:44:52,533 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:44:52,536 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:44:52,536 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:44:59,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.82 % 2021-12-02 04:44:59,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:44:59,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:45:02,549 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.82 % 2021-12-02 04:45:02,549 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 26.82 2021-12-02 04:45:02,575 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:45:02,578 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:45:02,578 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:45:09,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.82 % 2021-12-02 04:45:09,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:45:09,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:45:12,589 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.82 % 2021-12-02 04:45:12,589 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 26.82 2021-12-02 04:45:12,605 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:45:12,608 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:45:12,608 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:45:19,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.07 % 2021-12-02 04:45:19,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:45:19,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:45:22,619 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.07 % 2021-12-02 04:45:22,619 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 27.07 2021-12-02 04:45:22,634 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:45:22,638 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:45:22,638 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:45:29,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.07 % 2021-12-02 04:45:29,354 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:45:29,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:45:32,651 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.57 % 2021-12-02 04:45:32,651 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 27.57 2021-12-02 04:45:32,665 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:45:32,668 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:45:32,668 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:45:39,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.57 % 2021-12-02 04:45:39,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:45:39,374 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:45:42,680 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.57 % 2021-12-02 04:45:42,681 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 27.57 2021-12-02 04:45:42,697 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:45:42,700 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:45:42,700 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:45:49,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:45:49,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:45:49,392 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:45:52,713 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:45:52,713 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 27.81 2021-12-02 04:45:52,732 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:45:52,734 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:45:52,734 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:45:59,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:45:59,415 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:45:59,415 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:46:02,750 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:46:02,750 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 27.81 2021-12-02 04:46:02,766 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:46:02,767 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:46:02,767 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:46:09,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:46:09,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:46:09,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:46:12,779 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:46:12,779 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 27.81 2021-12-02 04:46:12,793 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:46:12,796 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:46:12,796 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:46:19,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:46:19,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:46:19,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:46:22,806 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:46:22,808 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 27.81 2021-12-02 04:46:22,825 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:46:22,828 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:46:22,828 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:46:29,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:46:29,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:46:29,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:46:32,842 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:46:32,842 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 27.81 2021-12-02 04:46:32,859 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:46:32,862 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:46:32,862 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:46:39,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:46:39,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:46:39,499 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:46:42,878 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:46:42,878 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 27.81 2021-12-02 04:46:42,894 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:46:42,898 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:46:42,898 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:46:49,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.81 % 2021-12-02 04:46:49,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:46:49,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:46:52,914 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.05 % 2021-12-02 04:46:52,914 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 28.05 2021-12-02 04:46:52,937 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:46:52,940 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:46:52,940 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:46:59,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.05 % 2021-12-02 04:46:59,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:46:59,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:47:02,953 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.30 % 2021-12-02 04:47:02,954 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 28.3 2021-12-02 04:47:02,971 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:47:02,973 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:47:02,973 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:47:09,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.30 % 2021-12-02 04:47:09,561 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:47:09,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:47:12,986 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.30 % 2021-12-02 04:47:12,986 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 28.3 2021-12-02 04:47:13,002 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:47:13,003 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:47:13,003 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:47:19,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.54 % 2021-12-02 04:47:19,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:47:19,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:47:23,016 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.54 % 2021-12-02 04:47:23,016 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 28.54 2021-12-02 04:47:23,030 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:47:23,032 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:47:23,032 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:47:29,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.54 % 2021-12-02 04:47:29,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:47:29,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:47:33,046 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.54 % 2021-12-02 04:47:33,046 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 28.54 2021-12-02 04:47:33,062 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:47:33,063 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:47:33,065 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:47:39,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.54 % 2021-12-02 04:47:39,630 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:47:39,630 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:47:43,075 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.54 % 2021-12-02 04:47:43,076 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 28.54 2021-12-02 04:47:43,089 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:47:43,092 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:47:43,092 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:47:49,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.54 % 2021-12-02 04:47:49,650 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:47:49,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:47:53,107 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.54 % 2021-12-02 04:47:53,107 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 28.54 2021-12-02 04:47:53,124 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:47:53,125 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:47:53,127 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:47:59,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.78 % 2021-12-02 04:47:59,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:47:59,668 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:48:03,138 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.78 % 2021-12-02 04:48:03,138 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 28.78 2021-12-02 04:48:03,155 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:48:03,157 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:48:03,157 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:48:09,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.78 % 2021-12-02 04:48:09,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:48:09,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:48:13,171 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.78 % 2021-12-02 04:48:13,171 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 28.78 2021-12-02 04:48:13,186 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:48:13,187 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:48:13,187 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:48:19,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.78 % 2021-12-02 04:48:19,706 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:48:19,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:48:23,200 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.78 % 2021-12-02 04:48:23,200 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 28.78 2021-12-02 04:48:23,220 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:48:23,223 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:48:23,223 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:48:29,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.03 % 2021-12-02 04:48:29,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:48:29,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:48:33,236 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.03 % 2021-12-02 04:48:33,236 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 29.03 2021-12-02 04:48:33,250 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:48:33,253 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:48:33,253 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:48:39,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.03 % 2021-12-02 04:48:39,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:48:39,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:48:43,266 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.03 % 2021-12-02 04:48:43,266 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 29.03 2021-12-02 04:48:43,286 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:48:43,290 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:48:43,290 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:48:49,765 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.03 % 2021-12-02 04:48:49,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:48:49,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:48:53,302 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.03 % 2021-12-02 04:48:53,302 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 29.03 2021-12-02 04:48:53,318 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:48:53,321 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:48:53,321 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:48:59,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.03 % 2021-12-02 04:48:59,795 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:48:59,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:49:03,334 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.03 % 2021-12-02 04:49:03,335 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 29.03 2021-12-02 04:49:03,351 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:49:03,354 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:49:03,354 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:49:09,806 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.51 % 2021-12-02 04:49:09,813 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:49:09,813 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:49:13,368 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.51 % 2021-12-02 04:49:13,368 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 29.51 2021-12-02 04:49:13,384 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:49:13,387 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:49:13,387 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:49:19,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.76 % 2021-12-02 04:49:19,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:49:19,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:49:23,401 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.76 % 2021-12-02 04:49:23,401 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 29.76 2021-12-02 04:49:23,417 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:49:23,420 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:49:23,421 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:49:29,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.76 % 2021-12-02 04:49:29,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:49:29,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:49:33,433 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.76 % 2021-12-02 04:49:33,433 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 29.76 2021-12-02 04:49:33,450 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:49:33,453 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:49:33,454 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:49:39,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.76 % 2021-12-02 04:49:39,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:49:39,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:49:43,469 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.76 % 2021-12-02 04:49:43,470 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 29.76 2021-12-02 04:49:43,486 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:49:43,490 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:49:43,490 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:49:49,892 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.76 % 2021-12-02 04:49:49,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:49:49,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:49:53,502 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.76 % 2021-12-02 04:49:53,502 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 29.76 2021-12-02 04:49:53,516 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:49:53,519 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:49:53,519 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:49:59,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.01 % 2021-12-02 04:49:59,918 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:49:59,920 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:50:03,532 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.01 % 2021-12-02 04:50:03,532 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 30.01 2021-12-02 04:50:03,546 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:50:03,549 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:50:03,549 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:50:09,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.01 % 2021-12-02 04:50:09,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:50:09,938 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:50:13,559 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.01 % 2021-12-02 04:50:13,561 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 30.01 2021-12-02 04:50:13,575 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:50:13,578 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:50:13,578 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:50:19,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.01 % 2021-12-02 04:50:19,954 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:50:19,954 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:50:23,592 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.01 % 2021-12-02 04:50:23,592 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 30.01 2021-12-02 04:50:23,605 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:50:23,608 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:50:23,608 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:50:29,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.51 % 2021-12-02 04:50:29,974 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:50:29,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:50:33,619 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.51 % 2021-12-02 04:50:33,619 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 30.51 2021-12-02 04:50:33,634 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:50:33,637 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:50:33,637 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:50:39,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.51 % 2021-12-02 04:50:39,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:50:39,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:50:43,650 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.51 % 2021-12-02 04:50:43,650 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 30.51 2021-12-02 04:50:43,664 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:50:43,667 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:50:43,667 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:50:50,007 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.74 % 2021-12-02 04:50:50,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:50:50,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:50:53,680 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.74 % 2021-12-02 04:50:53,680 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 30.74 2021-12-02 04:50:53,696 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:50:53,697 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:50:53,697 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:51:00,030 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.74 % 2021-12-02 04:51:00,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:51:00,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:51:03,713 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.74 % 2021-12-02 04:51:03,713 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 30.74 2021-12-02 04:51:03,727 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:51:03,730 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:51:03,730 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:51:10,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.74 % 2021-12-02 04:51:10,061 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:51:10,061 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:51:13,743 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.74 % 2021-12-02 04:51:13,743 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 30.74 2021-12-02 04:51:13,759 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:51:13,762 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:51:13,762 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:51:20,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.74 % 2021-12-02 04:51:20,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:51:20,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:51:23,776 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.74 % 2021-12-02 04:51:23,776 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 30.74 2021-12-02 04:51:23,796 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:51:23,799 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:51:23,799 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:51:30,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.74 % 2021-12-02 04:51:30,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:51:30,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:51:33,812 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.74 % 2021-12-02 04:51:33,812 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 30.74 2021-12-02 04:51:33,829 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:51:33,832 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:51:33,832 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:51:40,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.00 % 2021-12-02 04:51:40,114 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:51:40,114 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:51:43,845 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.00 % 2021-12-02 04:51:43,846 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 31.0 2021-12-02 04:51:43,862 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:51:43,865 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:51:43,865 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:51:50,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.00 % 2021-12-02 04:51:50,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:51:50,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:51:53,877 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.00 % 2021-12-02 04:51:53,878 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 31.0 2021-12-02 04:51:53,894 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:51:53,897 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:51:53,897 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:52:00,147 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.24 % 2021-12-02 04:52:00,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:52:00,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:52:03,908 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.24 % 2021-12-02 04:52:03,908 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 31.24 2021-12-02 04:52:03,924 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:52:03,927 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:52:03,927 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:52:10,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:52:10,170 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:52:10,170 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:52:13,938 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:52:13,940 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 31.73 2021-12-02 04:52:13,953 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:52:13,957 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:52:13,957 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:52:20,181 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:52:20,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:52:20,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:52:23,970 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:52:23,970 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 31.73 2021-12-02 04:52:23,986 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:52:23,989 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:52:23,989 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:52:30,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:52:30,203 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:52:30,204 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:52:34,000 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:52:34,000 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 31.73 2021-12-02 04:52:34,015 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:52:34,017 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:52:34,017 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:52:40,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:52:40,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:52:40,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:52:44,029 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:52:44,029 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 31.73 2021-12-02 04:52:44,046 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:52:44,048 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:52:44,049 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:52:50,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:52:50,244 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:52:50,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:52:54,062 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:52:54,062 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 31.73 2021-12-02 04:52:54,079 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:52:54,082 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:52:54,082 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:53:00,257 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:53:00,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:53:00,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:53:04,095 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:53:04,095 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 31.73 2021-12-02 04:53:04,112 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:53:04,115 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:53:04,115 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:53:10,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:53:10,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:53:10,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:53:14,131 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.73 % 2021-12-02 04:53:14,131 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 31.73 2021-12-02 04:53:14,154 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:53:14,157 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:53:14,157 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:53:20,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.98 % 2021-12-02 04:53:20,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:53:20,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:53:24,171 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.98 % 2021-12-02 04:53:24,171 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 31.98 2021-12-02 04:53:24,187 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:53:24,188 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:53:24,188 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:53:30,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.22 % 2021-12-02 04:53:30,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:53:30,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:53:34,201 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.22 % 2021-12-02 04:53:34,201 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 32.22 2021-12-02 04:53:34,217 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:53:34,219 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:53:34,219 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:53:40,346 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.47 % 2021-12-02 04:53:40,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:53:40,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:53:44,240 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.47 % 2021-12-02 04:53:44,242 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 32.47 2021-12-02 04:53:44,263 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:53:44,266 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:53:44,266 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:53:50,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.71 % 2021-12-02 04:53:50,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:53:50,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:53:54,279 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.71 % 2021-12-02 04:53:54,279 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 32.71 2021-12-02 04:53:54,302 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:53:54,305 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:53:54,319 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:54:00,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.71 % 2021-12-02 04:54:00,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:54:00,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:54:04,335 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.71 % 2021-12-02 04:54:04,335 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 32.71 2021-12-02 04:54:04,355 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:54:04,359 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:54:04,359 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:54:10,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.71 % 2021-12-02 04:54:10,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:54:10,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:54:14,374 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.71 % 2021-12-02 04:54:14,374 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 32.71 2021-12-02 04:54:14,390 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:54:14,392 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:54:14,392 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:54:20,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.71 % 2021-12-02 04:54:20,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:54:20,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:54:24,407 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.71 % 2021-12-02 04:54:24,407 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 32.71 2021-12-02 04:54:24,421 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:54:24,424 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:54:24,424 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:54:30,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.71 % 2021-12-02 04:54:30,469 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:54:30,469 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:54:34,437 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.71 % 2021-12-02 04:54:34,437 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 32.71 2021-12-02 04:54:34,453 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:54:34,456 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:54:34,456 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:54:40,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.71 % 2021-12-02 04:54:40,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:54:40,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:54:44,470 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.71 % 2021-12-02 04:54:44,470 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 32.71 2021-12-02 04:54:44,486 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:54:44,489 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:54:44,489 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:54:50,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.97 % 2021-12-02 04:54:50,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:54:50,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:54:54,500 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.97 % 2021-12-02 04:54:54,500 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 32.97 2021-12-02 04:54:54,515 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:54:54,517 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:54:54,517 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:55:00,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.20 % 2021-12-02 04:55:00,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:55:00,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:55:04,530 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.20 % 2021-12-02 04:55:04,530 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 33.2 2021-12-02 04:55:04,546 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:55:04,549 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:55:04,549 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:55:10,545 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:55:10,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:55:10,553 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:55:14,562 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:55:14,562 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 33.7 2021-12-02 04:55:14,578 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:55:14,581 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:55:14,581 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:55:20,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:55:20,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:55:20,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:55:24,594 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:55:24,594 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 33.7 2021-12-02 04:55:24,611 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:55:24,614 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:55:24,615 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:55:30,586 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:55:30,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:55:30,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:55:34,625 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:55:34,625 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 33.7 2021-12-02 04:55:34,641 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:55:34,644 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:55:34,644 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:55:40,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:55:40,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:55:40,612 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:55:44,658 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:55:44,658 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 33.7 2021-12-02 04:55:44,677 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:55:44,680 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:55:44,681 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:55:50,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:55:50,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:55:50,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:55:54,694 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:55:54,694 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 33.7 2021-12-02 04:55:54,710 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:55:54,713 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:55:54,713 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:56:00,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:56:00,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:56:00,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:56:04,724 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:56:04,726 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 33.7 2021-12-02 04:56:04,743 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:56:04,746 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:56:04,747 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:56:10,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:56:10,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:56:10,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:56:14,760 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.70 % 2021-12-02 04:56:14,762 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 33.7 2021-12-02 04:56:14,778 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:56:14,782 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:56:14,782 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:56:20,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.20 % 2021-12-02 04:56:20,696 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:56:20,696 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:56:24,796 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.20 % 2021-12-02 04:56:24,796 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 34.2 2021-12-02 04:56:24,816 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:56:24,821 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:56:24,821 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:56:30,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.20 % 2021-12-02 04:56:30,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:56:30,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:56:34,841 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.20 % 2021-12-02 04:56:34,841 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 34.2 2021-12-02 04:56:34,858 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:56:34,861 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:56:34,861 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:56:40,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.45 % 2021-12-02 04:56:40,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:56:40,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:56:44,874 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.45 % 2021-12-02 04:56:44,875 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 34.45 2021-12-02 04:56:44,888 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:56:44,891 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:56:44,891 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:56:50,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:56:50,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:56:50,755 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:56:54,901 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:56:54,903 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 34.69 2021-12-02 04:56:54,917 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:56:54,920 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:56:54,920 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:57:00,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:57:00,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:57:00,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:57:04,936 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:57:04,937 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 34.69 2021-12-02 04:57:04,954 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:57:04,957 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:57:04,957 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:57:10,783 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:57:10,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:57:10,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:57:14,970 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:57:14,970 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 34.69 2021-12-02 04:57:14,986 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:57:14,990 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:57:14,990 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:57:20,803 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:57:20,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:57:20,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:57:25,000 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:57:25,000 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 34.69 2021-12-02 04:57:25,013 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:57:25,016 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:57:25,016 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:57:30,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:57:30,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:57:30,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:57:35,029 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:57:35,029 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 34.69 2021-12-02 04:57:35,045 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:57:35,046 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:57:35,046 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:57:40,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:57:40,849 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:57:40,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:57:45,058 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:57:45,059 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 34.69 2021-12-02 04:57:45,073 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:57:45,076 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:57:45,076 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:57:50,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:57:50,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:57:50,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:57:55,089 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.69 % 2021-12-02 04:57:55,091 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 34.69 2021-12-02 04:57:55,105 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:57:55,108 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:57:55,108 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:58:00,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.19 % 2021-12-02 04:58:00,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:58:00,885 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:58:05,121 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.19 % 2021-12-02 04:58:05,121 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 35.19 2021-12-02 04:58:05,137 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:58:05,138 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:58:05,138 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:58:10,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.19 % 2021-12-02 04:58:10,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:58:10,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:58:15,151 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.19 % 2021-12-02 04:58:15,151 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 35.19 2021-12-02 04:58:15,164 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:58:15,167 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:58:15,167 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:58:20,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.44 % 2021-12-02 04:58:20,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:58:20,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:58:25,180 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.44 % 2021-12-02 04:58:25,180 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 35.44 2021-12-02 04:58:25,197 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:58:25,200 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:58:25,200 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:58:30,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.44 % 2021-12-02 04:58:30,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:58:30,946 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:58:35,213 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.69 % 2021-12-02 04:58:35,213 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 35.69 2021-12-02 04:58:35,233 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:58:35,234 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:58:35,234 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:58:40,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.69 % 2021-12-02 04:58:40,964 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:58:40,964 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:58:45,247 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.69 % 2021-12-02 04:58:45,247 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 35.69 2021-12-02 04:58:45,263 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:58:45,266 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:58:45,266 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:58:50,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.69 % 2021-12-02 04:58:50,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:58:50,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:58:55,278 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.69 % 2021-12-02 04:58:55,278 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 35.69 2021-12-02 04:58:55,292 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:58:55,296 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:58:55,296 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:59:00,999 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.69 % 2021-12-02 04:59:01,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:59:01,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:59:05,309 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.69 % 2021-12-02 04:59:05,309 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 35.69 2021-12-02 04:59:05,328 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:59:05,331 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:59:05,331 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:59:11,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.93 % 2021-12-02 04:59:11,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:59:11,030 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:59:15,345 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.93 % 2021-12-02 04:59:15,345 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 35.93 2021-12-02 04:59:15,365 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:59:15,368 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:59:15,368 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:59:21,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.93 % 2021-12-02 04:59:21,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:59:21,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:59:25,384 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.93 % 2021-12-02 04:59:25,384 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 35.93 2021-12-02 04:59:25,415 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:59:25,418 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:59:25,420 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:59:31,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.18 % 2021-12-02 04:59:31,079 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:59:31,079 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:59:35,431 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.18 % 2021-12-02 04:59:35,431 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 36.18 2021-12-02 04:59:35,448 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:59:35,450 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:59:35,450 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:59:41,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.18 % 2021-12-02 04:59:41,104 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:59:41,104 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:59:45,463 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.43 % 2021-12-02 04:59:45,463 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 36.43 2021-12-02 04:59:45,479 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:59:45,480 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:59:45,480 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:59:51,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.43 % 2021-12-02 04:59:51,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 04:59:51,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 04:59:55,493 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.43 % 2021-12-02 04:59:55,493 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 36.43 2021-12-02 04:59:55,509 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 04:59:55,510 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 04:59:55,512 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:00:01,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.43 % 2021-12-02 05:00:01,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:00:01,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:00:05,523 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.43 % 2021-12-02 05:00:05,523 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 36.43 2021-12-02 05:00:05,536 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:00:05,539 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:00:05,539 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:00:11,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.43 % 2021-12-02 05:00:11,165 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:00:11,165 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:00:15,551 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.43 % 2021-12-02 05:00:15,551 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 36.43 2021-12-02 05:00:15,565 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:00:15,568 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:00:15,569 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:00:21,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.67 % 2021-12-02 05:00:21,186 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:00:21,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:00:25,579 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.67 % 2021-12-02 05:00:25,581 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 36.67 2021-12-02 05:00:25,595 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:00:25,598 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:00:25,598 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:00:31,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.91 % 2021-12-02 05:00:31,204 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:00:31,204 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:00:35,609 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.91 % 2021-12-02 05:00:35,609 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 36.91 2021-12-02 05:00:35,625 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:00:35,627 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:00:35,628 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:00:41,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.91 % 2021-12-02 05:00:41,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:00:41,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:00:45,638 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.91 % 2021-12-02 05:00:45,638 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 36.91 2021-12-02 05:00:45,654 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:00:45,655 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:00:45,657 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:00:51,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.91 % 2021-12-02 05:00:51,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:00:51,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:00:55,674 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.91 % 2021-12-02 05:00:55,674 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 36.91 2021-12-02 05:00:55,690 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:00:55,693 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:00:55,693 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:01:01,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.91 % 2021-12-02 05:01:01,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:01:01,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:01:05,704 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.91 % 2021-12-02 05:01:05,704 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 36.91 2021-12-02 05:01:05,720 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:01:05,726 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:01:05,726 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:01:11,272 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.16 % 2021-12-02 05:01:11,278 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:01:11,278 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:01:15,736 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.16 % 2021-12-02 05:01:15,737 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 37.16 2021-12-02 05:01:15,752 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:01:15,755 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:01:15,755 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:01:21,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.16 % 2021-12-02 05:01:21,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:01:21,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:01:25,767 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.16 % 2021-12-02 05:01:25,767 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 37.16 2021-12-02 05:01:25,782 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:01:25,785 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:01:25,785 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:01:31,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.41 % 2021-12-02 05:01:31,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:01:31,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:01:35,796 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.66 % 2021-12-02 05:01:35,796 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 37.66 2021-12-02 05:01:35,811 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:01:35,812 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:01:35,813 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:01:41,336 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.66 % 2021-12-02 05:01:41,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:01:41,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:01:45,841 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.66 % 2021-12-02 05:01:45,842 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 37.66 2021-12-02 05:01:45,857 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:01:45,858 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:01:45,859 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:01:51,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.90 % 2021-12-02 05:01:51,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:01:51,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:01:55,871 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.90 % 2021-12-02 05:01:55,872 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 37.9 2021-12-02 05:01:55,888 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:01:55,891 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:01:55,891 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:02:01,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.90 % 2021-12-02 05:02:01,397 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:02:01,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:02:05,904 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.90 % 2021-12-02 05:02:05,904 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 37.9 2021-12-02 05:02:05,920 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:02:05,921 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:02:05,921 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:02:11,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.90 % 2021-12-02 05:02:11,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:02:11,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:02:15,936 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.90 % 2021-12-02 05:02:15,936 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 37.9 2021-12-02 05:02:15,951 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:02:15,954 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:02:15,956 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:02:21,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.90 % 2021-12-02 05:02:21,441 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:02:21,441 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:02:25,970 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.90 % 2021-12-02 05:02:25,970 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 37.9 2021-12-02 05:02:25,986 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:02:25,987 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:02:25,987 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:02:31,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.15 % 2021-12-02 05:02:31,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:02:31,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:02:36,000 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.15 % 2021-12-02 05:02:36,000 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 38.15 2021-12-02 05:02:36,017 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:02:36,020 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:02:36,020 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:02:41,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.15 % 2021-12-02 05:02:41,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:02:41,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:02:46,032 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.15 % 2021-12-02 05:02:46,032 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 38.15 2021-12-02 05:02:46,049 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:02:46,051 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:02:46,051 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:02:51,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.15 % 2021-12-02 05:02:51,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:02:51,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:02:56,063 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.15 % 2021-12-02 05:02:56,063 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 38.15 2021-12-02 05:02:56,078 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:02:56,081 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:02:56,081 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:03:01,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.15 % 2021-12-02 05:03:01,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:03:01,530 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:03:06,092 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.15 % 2021-12-02 05:03:06,092 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 38.15 2021-12-02 05:03:06,108 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:03:06,109 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:03:06,111 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:03:11,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.15 % 2021-12-02 05:03:11,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:03:11,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:03:16,125 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.40 % 2021-12-02 05:03:16,125 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 38.4 2021-12-02 05:03:16,141 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:03:16,144 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:03:16,144 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:03:21,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.64 % 2021-12-02 05:03:21,576 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:03:21,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:03:26,157 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.64 % 2021-12-02 05:03:26,157 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 38.64 2021-12-02 05:03:26,173 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:03:26,174 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:03:26,174 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:03:31,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.64 % 2021-12-02 05:03:31,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:03:31,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:03:36,187 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.88 % 2021-12-02 05:03:36,187 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 38.88 2021-12-02 05:03:36,203 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:03:36,206 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:03:36,206 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:03:41,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.13 % 2021-12-02 05:03:41,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:03:41,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:03:46,220 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.13 % 2021-12-02 05:03:46,220 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 39.13 2021-12-02 05:03:46,236 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:03:46,237 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:03:46,237 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:03:51,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.13 % 2021-12-02 05:03:51,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:03:51,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:03:56,253 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.13 % 2021-12-02 05:03:56,253 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 39.13 2021-12-02 05:03:56,272 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:03:56,273 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:03:56,273 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:04:01,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.13 % 2021-12-02 05:04:01,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:04:01,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:04:06,286 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.13 % 2021-12-02 05:04:06,286 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 39.13 2021-12-02 05:04:06,302 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:04:06,305 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:04:06,305 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:04:11,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.13 % 2021-12-02 05:04:11,681 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:04:11,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:04:16,318 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.13 % 2021-12-02 05:04:16,318 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 39.13 2021-12-02 05:04:16,338 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:04:16,341 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:04:16,341 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:04:21,691 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.13 % 2021-12-02 05:04:21,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:04:21,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:04:26,354 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.38 % 2021-12-02 05:04:26,355 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 39.38 2021-12-02 05:04:26,374 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:04:26,377 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:04:26,377 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:04:31,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.38 % 2021-12-02 05:04:31,719 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:04:31,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:04:36,390 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.38 % 2021-12-02 05:04:36,390 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 39.38 2021-12-02 05:04:36,404 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:04:36,407 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:04:36,408 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:04:41,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.38 % 2021-12-02 05:04:41,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:04:41,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:04:46,423 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.38 % 2021-12-02 05:04:46,424 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 39.38 2021-12-02 05:04:46,440 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:04:46,444 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:04:46,444 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:04:51,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.38 % 2021-12-02 05:04:51,762 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:04:51,762 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:04:56,457 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.63 % 2021-12-02 05:04:56,457 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 39.63 2021-12-02 05:04:56,471 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:04:56,473 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:04:56,473 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:05:01,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.87 % 2021-12-02 05:05:01,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:05:01,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:05:06,486 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.87 % 2021-12-02 05:05:06,486 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 39.87 2021-12-02 05:05:06,502 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:05:06,505 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:05:06,505 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:05:11,795 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.12 % 2021-12-02 05:05:11,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:05:11,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:05:16,516 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.12 % 2021-12-02 05:05:16,516 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 40.12 2021-12-02 05:05:16,532 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:05:16,535 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:05:16,535 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:05:21,805 | test | INFO | MainThread | [rest_client:rebalance_reached:119] Rebalance reached >40.12% in 3557.39700007 seconds 2021-12-02 05:05:21,816 | test | DEBUG | MainThread | [rest_client:diag_eval:1453] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-02 05:05:21,816 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-02 05:05:21,819 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 05:05:21,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 05:05:22,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 05:05:22,158 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 05:05:22,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 05:05:22,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 05:05:22,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 05:05:22,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 05:05:22,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 05:05:22,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 05:05:22,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 05:05:22,260 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 05:05:22,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 05:05:22,306 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['280219 memcached\n'], []) 2021-12-02 05:05:22,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 05:05:22,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 05:05:22,426 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['280219 memcached\n'], []) 2021-12-02 05:05:22,426 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 05:05:22,428 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 05:05:22,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 05:05:22,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 05:05:22,766 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 05:05:22,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 05:05:22,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 05:05:22,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 05:05:22,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 05:05:22,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 05:05:22,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 05:05:22,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 05:05:22,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 05:05:22,868 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 05:05:22,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 05:05:22,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:22,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:05:22,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:05:23,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:05:23,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:23,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:05:23,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 05:05:23,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 05:05:23,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:23,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:05:23,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 05:05:23,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 05:05:23,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 05:05:23,209 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 05:05:23,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 05:05:23,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:23,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 05:05:23,240 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 05:05:23,240 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 05:05:23,246 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 05:05:23,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 05:05:23,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 05:05:23,746 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 05:05:23,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 05:05:23,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 05:05:23,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 05:05:23,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 05:05:23,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 05:05:23,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 05:05:23,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 05:05:23,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 05:05:23,855 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 05:05:23,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 05:05:23,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:23,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:05:23,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:05:23,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:05:23,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 05:05:24,009 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 05:05:24,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 05:05:24,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:24,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 05:05:24,046 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 05:05:24,046 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 05:05:24,049 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 05:05:24,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 05:05:24,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 05:05:24,463 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 05:05:24,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 05:05:24,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 05:05:24,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 05:05:24,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 05:05:24,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 05:05:24,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 05:05:24,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 05:05:24,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 05:05:24,614 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 05:05:24,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 05:05:24,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:24,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:05:24,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:05:24,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:05:24,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:24,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:05:24,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 05:05:24,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 05:05:24,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:24,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:05:25,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 05:05:25,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 05:05:25,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:25,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:05:25,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 05:05:25,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 05:05:25,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:25,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:05:25,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 05:05:25,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 05:05:25,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 05:05:25,348 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 05:05:25,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 05:05:25,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:25,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 05:05:25,404 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 05:05:25,404 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 05:05:25,407 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 05:05:25,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 05:05:25,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 05:05:25,898 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 05:05:25,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 05:05:25,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 05:05:25,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 05:05:25,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 05:05:25,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 05:05:25,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 05:05:26,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 05:05:26,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 05:05:26,035 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 05:05:26,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 05:05:26,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:26,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:05:26,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:05:26,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:05:26,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:26,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:05:26,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 05:05:26,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 05:05:26,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:26,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:05:26,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 05:05:26,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 05:05:26,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 05:05:26,548 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.12 % 2021-12-02 05:05:26,549 | test | INFO | pool-3-thread-21 | [task:check:384] Rebalance - status: running, progress: 40.12 2021-12-02 05:05:26,552 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 05:05:26,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 05:05:26,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:05:26,569 | infra | DEBUG | pool-3-thread-21 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:05:26,571 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:05:26,571 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:05:26,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 05:05:26,599 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 05:05:26,599 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 05:05:26,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-02 05:05:26,608 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-02 05:05:36,582 | test | ERROR | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1640] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'7b674fce6a66eba072f87dbc5b6da447', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=b9cdd48f20f934fadbcc3c7a22dd4e20', u'status': u'notRunning'} - rebalance failed 2021-12-02 05:05:36,604 | test | INFO | pool-3-thread-21 | [rest_client:print_UI_logs:2789] Latest logs from UI on 172.23.110.65: 2021-12-02 05:05:36,604 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638450332912L, u'shortText': u'message', u'serverTime': u'2021-12-02T05:05:32.912Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.25374.117>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,119,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.24541.111>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 119}},\n infinity]}}}}}.\nRebalance Operation Id = 123bd92a27793fac1899a0e32eb2c05f'} 2021-12-02 05:05:36,604 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638450332886L, u'shortText': u'message', u'serverTime': u'2021-12-02T05:05:32.886Z', u'text': u'Worker <0.25346.117> (for action {move,\n {119,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.65\'],\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.25374.117>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 151}]},\n {mc_client_binary,\n select_bucket,\n 2,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 323}]},\n {ns_memcached_sockets_pool,\n take_socket,\n 2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 82}]},\n {async,\n \'-async_init/4-fun-1-\',\n 3,\n [{file,\n "src/async.erl"},\n {line,\n 191}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 119,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.24541.111>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 119}},\n infinity]}}}}'} 2021-12-02 05:05:36,605 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638450332876L, u'shortText': u'message', u'serverTime': u'2021-12-02T05:05:32.876Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1202 03:55:22.727381 280448 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-02 05:05:36,605 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638450332875L, u'shortText': u'message', u'serverTime': u'2021-12-02T05:05:32.875Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-02 05:05:36,605 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638446644797L, u'shortText': u'message', u'serverTime': u'2021-12-02T04:04:04.797Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-02 05:05:36,605 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638446644647L, u'shortText': u'message', u'serverTime': u'2021-12-02T04:04:04.647Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-02 05:05:36,605 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638446644420L, u'shortText': u'message', u'serverTime': u'2021-12-02T04:04:04.420Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.68',\n 'ns_1@172.23.110.66','ns_1@172.23.110.65'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 123bd92a27793fac1899a0e32eb2c05f"} 2021-12-02 05:05:36,605 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638446625827L, u'shortText': u'message', u'serverTime': u'2021-12-02T04:03:45.827Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 482 seconds.'} 2021-12-02 05:05:36,605 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638446114182L, u'shortText': u'message', u'serverTime': u'2021-12-02T03:55:14.182Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nW1201 20:33:57.140372 244731 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1201 21:53:53.328923 241683 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1201 23:20:14.538772 241640 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1202 01:27:17.196750 204658 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1202 02:48:05.196605 204661 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-02 05:05:36,605 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638446114175L, u'shortText': u'message', u'serverTime': u'2021-12-02T03:55:14.175Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.25989.110>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,stats_recv,4,\n [{file,"src/mc_client_binary.erl"},\n {line,163}]},\n {mc_client_binary,stats,4,\n [{file,"src/mc_client_binary.erl"},\n {line,455}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,475}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,224}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,322,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.31128.106>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.68\'],\n 322}},\n infinity]}}}}}.\nRebalance Operation Id = 4cc1039edf9bfabdf54520d56b8099ab'} 2021-12-02 05:05:36,605 | test | ERROR | pool-3-thread-21 | [task:call:296] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'7b674fce6a66eba072f87dbc5b6da447', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=b9cdd48f20f934fadbcc3c7a22dd4e20', u'status': u'notRunning'} - rebalance failed 2021-12-02 05:06:26,621 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-02 05:06:26,621 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 05:06:26,635 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-02 05:06:26,648 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.68:8091 2021-12-02 05:06:26,648 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 05:06:26,661 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.68:8091 is running 2021-12-02 05:06:26,674 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-02 05:06:26,674 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 05:06:26,684 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:26,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:26,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:28,703 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:28,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:28,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:30,726 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:30,732 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:30,732 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:32,743 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:32,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:32,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:34,763 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:34,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:34,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:36,789 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:36,795 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:36,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:38,808 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:38,816 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:38,816 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:40,832 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:40,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:40,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:42,855 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:42,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:42,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:44,875 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:44,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:44,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:46,897 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:46,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:46,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:48,923 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:48,931 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:48,931 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:50,947 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:50,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:50,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:52,976 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:52,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:52,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:55,000 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:55,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:55,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:57,023 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:57,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:57,030 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:06:59,043 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:06:59,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:06:59,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:01,066 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:01,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:01,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:03,086 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:03,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:03,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:05,111 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:05,117 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:05,117 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:07,131 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:07,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:07,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:09,148 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:09,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:09,155 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:11,170 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:11,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:11,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:13,190 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:13,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:13,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:15,213 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:15,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:15,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:17,233 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:17,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:17,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:19,253 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:19,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:19,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:21,273 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:21,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:21,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:23,295 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:23,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:23,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:25,315 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:25,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:25,323 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:27,335 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:27,344 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:27,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:29,357 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:29,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:29,364 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:31,375 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:31,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:31,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:33,394 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:33,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:33,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:35,415 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:35,423 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:35,423 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:37,434 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:37,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:37,441 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:39,453 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:39,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:39,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:41,474 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:41,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:41,482 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:43,513 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:43,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:43,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:45,536 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:45,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:45,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:47,558 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:47,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:47,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:49,575 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:49,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:49,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:51,595 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:51,602 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:51,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:53,617 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:53,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:53,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:55,638 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:55,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:55,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:57,658 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:57,665 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:57,665 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:07:59,677 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:07:59,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:07:59,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:01,697 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:01,706 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:01,706 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:03,717 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:03,724 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:03,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:05,739 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:05,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:05,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:07,759 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:07,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:07,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:09,779 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:09,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:09,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:11,798 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:11,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:11,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:13,819 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:13,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:13,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:15,838 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:15,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:15,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:17,858 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:17,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:17,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:19,881 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:19,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:19,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:21,904 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:21,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:21,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:23,926 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:23,933 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:23,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:25,947 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:25,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:25,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:27,967 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:27,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:27,974 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:29,990 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:29,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:29,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:32,013 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:32,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:32,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:34,038 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:34,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:34,045 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:36,056 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:36,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:36,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:38,078 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:38,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:38,086 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:40,101 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:40,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:40,109 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:42,121 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:42,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:42,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:44,141 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:44,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:44,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:46,164 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:46,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:46,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:48,187 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:48,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:48,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:50,207 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:50,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:50,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:52,230 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:52,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:52,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:54,252 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:54,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:54,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:56,273 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:56,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:56,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:08:58,292 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:08:58,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:08:58,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:00,316 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:00,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:00,323 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:02,336 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:02,344 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:02,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:04,358 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:04,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:04,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:06,378 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:06,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:06,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:08,411 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:08,418 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:08,418 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:10,433 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:10,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:10,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:12,456 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:12,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:12,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:14,476 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:14,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:14,484 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:16,497 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:16,505 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:16,505 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:18,516 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:18,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:18,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:20,536 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:20,543 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:20,543 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:22,559 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:22,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:22,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:24,967 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:24,974 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:24,974 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:26,986 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:26,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:26,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:29,006 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:29,012 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:29,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:31,026 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:31,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:31,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:33,045 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:33,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:33,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:35,065 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:35,071 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:35,071 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:37,085 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:37,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:37,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:39,111 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:39,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:39,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:41,137 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:41,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:41,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:43,164 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:43,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:43,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:45,184 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:45,191 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:45,191 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:47,206 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:47,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:47,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:49,226 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:49,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:49,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:51,249 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:51,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:51,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:53,270 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:53,278 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:53,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:55,292 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:55,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:55,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:57,312 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:57,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:57,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:09:59,334 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:09:59,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:09:59,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:01,355 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:01,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:01,362 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:03,378 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:03,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:03,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:05,407 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:05,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:05,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:07,428 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:07,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:07,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:09,450 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:09,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:09,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:11,470 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:11,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:11,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:13,492 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:13,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:13,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:15,510 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:15,517 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:15,517 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:17,536 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:17,543 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:17,543 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:19,558 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:19,563 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:19,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:21,578 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:21,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:21,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:23,601 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:23,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:23,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:25,625 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:25,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:25,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:27,642 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:27,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:27,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:29,661 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:29,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:29,668 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:31,683 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:31,688 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:31,688 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:33,700 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:33,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:33,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:35,720 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:35,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:35,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:37,743 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:37,749 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:37,749 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:39,760 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:39,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:39,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:41,782 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:41,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:41,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:43,802 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:43,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:43,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:45,822 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:45,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:45,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:47,844 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:47,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:47,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:49,865 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:49,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:49,872 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:51,885 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:51,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:51,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:53,905 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:53,913 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:53,913 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:55,924 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:55,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:55,931 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:57,944 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:57,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:57,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:10:59,964 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:10:59,971 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:10:59,971 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:01,986 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:01,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:01,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:04,006 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:04,013 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:04,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:06,026 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:06,035 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:06,035 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:08,049 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:08,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:08,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:10,069 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:10,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:10,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:12,088 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:12,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:12,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:14,108 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:14,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:14,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:16,131 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:16,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:16,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:18,154 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:18,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:18,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:20,171 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:20,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:20,178 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:22,193 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:22,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:22,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:24,213 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:24,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:24,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:26,234 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:26,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:26,242 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:28,256 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:28,262 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:28,262 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:30,275 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:30,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:30,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:32,292 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:32,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:32,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:34,312 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:34,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:34,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:36,332 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:36,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:36,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:38,351 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:38,357 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:38,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:40,372 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:40,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:40,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:42,398 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:42,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:42,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:44,420 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:44,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:44,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:46,440 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:46,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:46,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:48,460 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:48,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:48,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:52,039 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:52,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:52,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:54,061 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:54,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:54,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:56,082 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:56,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:56,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:11:58,101 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:11:58,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:11:58,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:00,122 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:00,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:00,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:02,144 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:02,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:02,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:04,167 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:04,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:04,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:06,190 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:06,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:06,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:08,213 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:08,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:08,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:10,234 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:10,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:10,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:12,256 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:12,262 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:12,262 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:14,273 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:14,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:14,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:16,292 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:16,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:16,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:18,312 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:18,318 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:18,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:20,332 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:20,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:20,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:22,352 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:22,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:22,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:24,372 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:24,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:24,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:26,394 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:26,400 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:26,400 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:28,414 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:28,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:28,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:30,434 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:30,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:30,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:32,456 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:32,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:32,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:34,476 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:34,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:34,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:36,494 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:36,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:36,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:38,516 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:38,522 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:38,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:40,535 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:40,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:40,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:42,558 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:42,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:42,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:44,582 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:44,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:44,589 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:46,604 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:46,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:46,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:48,625 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:48,632 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:48,632 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:50,647 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:50,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:50,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:52,667 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:52,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:52,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:54,687 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:54,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:54,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:56,707 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:56,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:56,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:12:58,730 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:12:58,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:12:58,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:13:00,753 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:13:00,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:13:00,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:13:02,773 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:13:02,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:13:02,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:13:04,793 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 05:13:04,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 05:13:04,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 05:13:06,826 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-02 05:13:06,844 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-02 05:13:06,844 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 05:13:06,858 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-02 05:13:06,861 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 05:13:06,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 05:13:07,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 05:13:07,184 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 05:13:07,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 05:13:07,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 05:13:07,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 05:13:07,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 05:13:07,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 05:13:07,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 05:13:07,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 05:13:07,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 05:13:07,302 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 05:13:07,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 05:13:07,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:07,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:07,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:13:07,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:13:07,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:07,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:07,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 05:13:07,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 05:13:07,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:07,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:07,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 05:13:07,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 05:13:07,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 05:13:07,673 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 05:13:07,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 05:13:07,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:07,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 05:13:07,710 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 05:13:07,710 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 05:13:07,716 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 05:13:07,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 05:13:08,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 05:13:08,085 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 05:13:08,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 05:13:08,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 05:13:08,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 05:13:08,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 05:13:08,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 05:13:08,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 05:13:08,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 05:13:08,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 05:13:08,213 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 05:13:08,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 05:13:08,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:08,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:08,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:13:08,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:13:08,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 05:13:08,404 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 05:13:08,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 05:13:08,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:08,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 05:13:08,453 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 05:13:08,453 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 05:13:08,457 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 05:13:08,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 05:13:08,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 05:13:08,882 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 05:13:08,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 05:13:08,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 05:13:08,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 05:13:08,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 05:13:08,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 05:13:08,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 05:13:09,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 05:13:09,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 05:13:09,035 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 05:13:09,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 05:13:09,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:09,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:09,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:13:09,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:13:09,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:09,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:09,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 05:13:09,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 05:13:09,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:09,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:09,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 05:13:09,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 05:13:09,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:09,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:09,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 05:13:09,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 05:13:09,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:09,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:09,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 05:13:09,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 05:13:09,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:09,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:09,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 05:13:09,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 05:13:09,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 05:13:09,785 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 05:13:09,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 05:13:09,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:09,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 05:13:09,826 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 05:13:09,826 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 05:13:09,829 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 05:13:09,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 05:13:10,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 05:13:10,219 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 05:13:10,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 05:13:10,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 05:13:10,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 05:13:10,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 05:13:10,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 05:13:10,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 05:13:10,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 05:13:10,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 05:13:10,345 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 05:13:10,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 05:13:10,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:10,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:10,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:13:10,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 05:13:10,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:10,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:10,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 05:13:10,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 05:13:10,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:10,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 05:13:10,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 05:13:10,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 05:13:10,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 05:13:10,746 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 05:13:10,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 05:13:10,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 05:13:10,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 05:13:10,788 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 05:13:10,789 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 05:13:10,789 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1638446644.36 2021-12-02 05:13:10,789 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 40 2021-12-02 05:13:10,828 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1638450790.79 2021-12-02 05:13:10,829 | infra | INFO | pool-3-thread-22 | [task:start_task:89] Thread 'Rebalance_task_IN=[]_OUT=[]_1638450790.79' started 2021-12-02 05:13:10,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-02 05:13:10,841 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-02 05:13:10,851 | test | DEBUG | pool-3-thread-22 | [rest_client:rebalance:1431] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-02 05:13:10,907 | test | DEBUG | pool-3-thread-22 | [rest_client:rebalance:1436] Rebalance operation started 2021-12-02 05:13:10,913 | test | INFO | pool-3-thread-22 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1817-enterprise | 2.61801259544 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1817-enterprise | 1.15356284138 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1817-enterprise | 3.33644196373 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1817-enterprise | 3.41601937693 | Cluster node | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-02 05:13:10,927 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 05:13:10,928 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 05:13:10,953 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:13:10,957 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:13:10,957 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:13:20,970 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 05:13:20,970 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 05:13:20,986 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:13:20,990 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:13:20,990 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:13:31,000 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 05:13:31,000 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 05:13:31,016 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:13:31,019 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:13:31,019 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:13:41,032 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 05:13:41,032 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 05:13:41,048 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:13:41,051 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:13:41,051 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:13:51,062 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 05:13:51,062 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 05:13:51,076 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:13:51,078 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:13:51,078 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:14:01,088 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 05:14:01,089 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 05:14:01,104 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:14:01,105 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:14:01,107 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:14:11,118 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 05:14:11,118 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 05:14:11,132 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:14:11,134 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:14:11,134 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:14:21,147 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.47 % 2021-12-02 05:14:21,148 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.47 2021-12-02 05:14:21,167 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:14:21,170 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:14:21,170 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:14:31,180 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.83 % 2021-12-02 05:14:31,180 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.83 2021-12-02 05:14:31,194 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:14:31,197 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:14:31,197 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:14:41,209 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.83 % 2021-12-02 05:14:41,209 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.83 2021-12-02 05:14:41,221 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:14:41,223 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:14:41,223 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:14:51,236 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.83 % 2021-12-02 05:14:51,236 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.83 2021-12-02 05:14:51,252 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:14:51,253 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:14:51,253 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:15:01,265 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.71 % 2021-12-02 05:15:01,266 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 1.71 2021-12-02 05:15:01,282 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:15:01,283 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:15:01,283 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:15:10,854 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.71 % 2021-12-02 05:15:10,857 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 1.71 2021-12-02 05:15:10,867 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.71 % 2021-12-02 05:15:10,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:15:10,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:15:11,298 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.71 % 2021-12-02 05:15:11,299 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 1.71 2021-12-02 05:15:11,316 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:15:11,319 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:15:11,319 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:15:20,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.71 % 2021-12-02 05:15:20,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:15:20,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:15:21,331 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.71 % 2021-12-02 05:15:21,332 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 1.71 2021-12-02 05:15:21,348 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:15:21,349 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:15:21,349 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:15:30,910 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.71 % 2021-12-02 05:15:30,918 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:15:30,918 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:15:31,361 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.71 % 2021-12-02 05:15:31,362 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 1.71 2021-12-02 05:15:31,381 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:15:31,384 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:15:31,384 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:15:40,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.07 % 2021-12-02 05:15:40,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:15:40,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:15:41,395 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.07 % 2021-12-02 05:15:41,397 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 2.07 2021-12-02 05:15:41,411 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:15:41,414 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:15:41,414 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:15:50,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.07 % 2021-12-02 05:15:50,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:15:50,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:15:51,427 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.07 % 2021-12-02 05:15:51,427 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 2.07 2021-12-02 05:15:51,447 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:15:51,450 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:15:51,450 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:16:00,974 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.49 % 2021-12-02 05:16:00,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:16:00,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:16:01,460 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.49 % 2021-12-02 05:16:01,461 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 2.49 2021-12-02 05:16:01,480 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:16:01,483 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:16:01,483 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:16:10,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.49 % 2021-12-02 05:16:11,005 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:16:11,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:16:11,496 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.49 % 2021-12-02 05:16:11,496 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 2.49 2021-12-02 05:16:11,516 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:16:11,519 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:16:11,519 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:16:21,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.95 % 2021-12-02 05:16:21,022 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:16:21,022 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:16:21,532 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.95 % 2021-12-02 05:16:21,532 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 2.95 2021-12-02 05:16:21,552 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:16:21,555 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:16:21,555 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:16:31,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.95 % 2021-12-02 05:16:31,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:16:31,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:16:31,566 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.95 % 2021-12-02 05:16:31,568 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 2.95 2021-12-02 05:16:31,584 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:16:31,586 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:16:31,586 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:16:41,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.42 % 2021-12-02 05:16:41,056 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:16:41,058 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:16:41,598 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.42 % 2021-12-02 05:16:41,598 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 3.42 2021-12-02 05:16:41,614 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:16:41,615 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:16:41,615 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:16:51,073 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.78 % 2021-12-02 05:16:51,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:16:51,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:16:51,627 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.78 % 2021-12-02 05:16:51,627 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 3.78 2021-12-02 05:16:51,641 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:16:51,644 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:16:51,644 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:17:01,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.78 % 2021-12-02 05:17:01,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:17:01,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:17:01,655 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.78 % 2021-12-02 05:17:01,655 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 3.78 2021-12-02 05:17:01,676 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:17:01,678 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:17:01,678 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:17:11,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.78 % 2021-12-02 05:17:11,127 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:17:11,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:17:11,690 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.78 % 2021-12-02 05:17:11,690 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 3.78 2021-12-02 05:17:11,707 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:17:11,710 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:17:11,710 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:17:21,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.78 % 2021-12-02 05:17:21,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:17:21,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:17:21,723 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.20 % 2021-12-02 05:17:21,723 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 4.2 2021-12-02 05:17:21,737 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:17:21,740 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:17:21,740 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:17:31,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.20 % 2021-12-02 05:17:31,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:17:31,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:17:31,750 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.20 % 2021-12-02 05:17:31,752 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 4.2 2021-12-02 05:17:31,767 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:17:31,772 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:17:31,772 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:17:41,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.20 % 2021-12-02 05:17:41,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:17:41,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:17:41,785 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.20 % 2021-12-02 05:17:41,785 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 4.2 2021-12-02 05:17:41,801 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:17:41,803 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:17:41,803 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:17:51,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.20 % 2021-12-02 05:17:51,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:17:51,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:17:51,815 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.20 % 2021-12-02 05:17:51,816 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 4.2 2021-12-02 05:17:51,834 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:17:51,836 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:17:51,838 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:18:01,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.61 % 2021-12-02 05:18:01,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:18:01,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:18:01,851 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.61 % 2021-12-02 05:18:01,852 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 4.61 2021-12-02 05:18:01,871 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:18:01,874 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:18:01,875 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:18:11,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.44 % 2021-12-02 05:18:11,232 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:18:11,232 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:18:11,884 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.44 % 2021-12-02 05:18:11,885 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 5.44 2021-12-02 05:18:11,901 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:18:11,904 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:18:11,904 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:18:21,242 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.44 % 2021-12-02 05:18:21,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:18:21,247 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:18:21,917 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.44 % 2021-12-02 05:18:21,917 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 5.44 2021-12-02 05:18:21,933 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:18:21,936 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:18:21,936 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:18:31,257 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.44 % 2021-12-02 05:18:31,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:18:31,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:18:31,953 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.44 % 2021-12-02 05:18:31,953 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 5.44 2021-12-02 05:18:31,969 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:18:31,971 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:18:31,971 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:18:41,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.44 % 2021-12-02 05:18:41,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:18:41,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:18:41,983 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.44 % 2021-12-02 05:18:41,983 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 5.44 2021-12-02 05:18:41,997 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:18:42,000 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:18:42,002 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:18:51,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.90 % 2021-12-02 05:18:51,298 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:18:51,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:18:52,012 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.90 % 2021-12-02 05:18:52,013 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 5.9 2021-12-02 05:18:52,026 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:18:52,029 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:18:52,029 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:19:01,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.90 % 2021-12-02 05:19:01,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:19:01,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:19:02,040 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.90 % 2021-12-02 05:19:02,040 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 5.9 2021-12-02 05:19:02,072 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:19:02,073 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:19:02,073 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:19:11,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.90 % 2021-12-02 05:19:11,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:19:11,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:19:12,085 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.90 % 2021-12-02 05:19:12,086 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 5.9 2021-12-02 05:19:12,105 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:19:12,108 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:19:12,108 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:19:21,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.32 % 2021-12-02 05:19:21,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:19:21,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:19:22,119 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.32 % 2021-12-02 05:19:22,119 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 6.32 2021-12-02 05:19:22,134 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:19:22,137 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:19:22,137 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:19:31,364 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.32 % 2021-12-02 05:19:31,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:19:31,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:19:32,148 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.73 % 2021-12-02 05:19:32,148 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 6.73 2021-12-02 05:19:32,161 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:19:32,164 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:19:32,164 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:19:41,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.73 % 2021-12-02 05:19:41,385 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:19:41,385 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:19:42,177 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.73 % 2021-12-02 05:19:42,177 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 6.73 2021-12-02 05:19:42,191 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:19:42,194 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:19:42,194 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:19:51,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.10 % 2021-12-02 05:19:51,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:19:51,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:19:52,204 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.10 % 2021-12-02 05:19:52,206 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 7.1 2021-12-02 05:19:52,220 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:19:52,223 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:19:52,223 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:20:01,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.10 % 2021-12-02 05:20:01,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:20:01,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:20:02,234 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.10 % 2021-12-02 05:20:02,234 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 7.1 2021-12-02 05:20:02,247 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:20:02,250 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:20:02,250 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:20:11,431 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.10 % 2021-12-02 05:20:11,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:20:11,438 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:20:12,260 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.56 % 2021-12-02 05:20:12,260 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 7.56 2021-12-02 05:20:12,275 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:20:12,276 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:20:12,276 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:20:21,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.56 % 2021-12-02 05:20:21,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:20:21,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:20:22,286 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.56 % 2021-12-02 05:20:22,288 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 7.56 2021-12-02 05:20:22,301 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:20:22,302 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:20:22,303 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:20:31,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.56 % 2021-12-02 05:20:31,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:20:31,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:20:32,313 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.56 % 2021-12-02 05:20:32,313 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 7.56 2021-12-02 05:20:32,328 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:20:32,331 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:20:32,331 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:20:41,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.56 % 2021-12-02 05:20:41,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:20:41,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:20:42,342 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.56 % 2021-12-02 05:20:42,342 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 7.56 2021-12-02 05:20:42,357 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:20:42,358 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:20:42,358 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:20:51,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.56 % 2021-12-02 05:20:51,522 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:20:51,522 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:20:52,369 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.56 % 2021-12-02 05:20:52,369 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 7.56 2021-12-02 05:20:52,391 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:20:52,392 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:20:52,392 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:21:01,533 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.03 % 2021-12-02 05:21:01,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:21:01,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:21:02,404 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.03 % 2021-12-02 05:21:02,404 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 8.03 2021-12-02 05:21:02,417 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:21:02,420 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:21:02,420 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:21:11,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.81 % 2021-12-02 05:21:11,563 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:21:11,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:21:12,433 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.81 % 2021-12-02 05:21:12,433 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 8.81 2021-12-02 05:21:12,450 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:21:12,451 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:21:12,451 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:21:21,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.81 % 2021-12-02 05:21:21,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:21:21,599 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:21:22,463 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.81 % 2021-12-02 05:21:22,463 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 8.81 2021-12-02 05:21:22,480 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:21:22,482 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:21:22,482 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:21:31,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.81 % 2021-12-02 05:21:31,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:21:31,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:21:32,496 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.81 % 2021-12-02 05:21:32,496 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 8.81 2021-12-02 05:21:32,517 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:21:32,520 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:21:32,522 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:21:41,637 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.81 % 2021-12-02 05:21:41,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:21:41,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:21:42,532 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.81 % 2021-12-02 05:21:42,533 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 8.81 2021-12-02 05:21:42,553 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:21:42,558 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:21:42,558 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:21:51,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.81 % 2021-12-02 05:21:51,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:21:51,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:21:52,569 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.81 % 2021-12-02 05:21:52,569 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 8.81 2021-12-02 05:21:52,589 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:21:52,592 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:21:52,592 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:22:01,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.22 % 2021-12-02 05:22:01,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:22:01,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:22:02,621 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.22 % 2021-12-02 05:22:02,622 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 9.22 2021-12-02 05:22:02,640 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:22:02,642 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:22:02,642 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:22:11,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.22 % 2021-12-02 05:22:11,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:22:11,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:22:12,654 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.22 % 2021-12-02 05:22:12,654 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 9.22 2021-12-02 05:22:12,668 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:22:12,671 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:22:12,671 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:22:21,709 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.68 % 2021-12-02 05:22:21,719 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:22:21,719 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:22:22,684 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.68 % 2021-12-02 05:22:22,684 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 9.68 2021-12-02 05:22:22,698 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:22:22,701 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:22:22,701 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:22:31,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.05 % 2021-12-02 05:22:31,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:22:31,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:22:32,713 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.05 % 2021-12-02 05:22:32,713 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 10.05 2021-12-02 05:22:32,727 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:22:32,730 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:22:32,730 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:22:41,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:22:41,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:22:41,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:22:42,740 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:22:42,740 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 10.51 2021-12-02 05:22:42,756 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:22:42,757 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:22:42,757 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:22:51,767 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:22:51,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:22:51,778 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:22:52,773 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:22:52,773 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 10.51 2021-12-02 05:22:52,796 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:22:52,803 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:22:52,803 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:23:01,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:23:01,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:23:01,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:23:02,818 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:23:02,818 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 10.51 2021-12-02 05:23:02,835 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:23:02,838 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:23:02,838 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:23:11,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:23:11,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:23:11,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:23:12,851 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:23:12,851 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 10.51 2021-12-02 05:23:12,867 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:23:12,869 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:23:12,869 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:23:21,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:23:21,839 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:23:21,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:23:22,881 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:23:22,881 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 10.51 2021-12-02 05:23:22,898 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:23:22,900 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:23:22,900 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:23:31,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:23:31,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:23:31,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:23:32,911 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:23:32,911 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 10.51 2021-12-02 05:23:32,927 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:23:32,930 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:23:32,930 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:23:41,872 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.51 % 2021-12-02 05:23:41,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:23:41,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:23:42,941 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.93 % 2021-12-02 05:23:42,943 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 10.93 2021-12-02 05:23:42,960 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:23:42,963 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:23:42,963 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:23:51,892 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.70 % 2021-12-02 05:23:51,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:23:51,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:23:52,976 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.70 % 2021-12-02 05:23:52,976 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 11.7 2021-12-02 05:23:52,992 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:23:52,996 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:23:52,996 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:24:01,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.70 % 2021-12-02 05:24:01,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:24:01,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:24:03,007 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.70 % 2021-12-02 05:24:03,009 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 11.7 2021-12-02 05:24:03,023 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:24:03,026 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:24:03,028 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:24:11,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.17 % 2021-12-02 05:24:11,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:24:11,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:24:13,039 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.17 % 2021-12-02 05:24:13,039 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 12.17 2021-12-02 05:24:13,055 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:24:13,059 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:24:13,059 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:24:21,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.17 % 2021-12-02 05:24:21,959 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:24:21,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:24:23,072 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.17 % 2021-12-02 05:24:23,072 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 12.17 2021-12-02 05:24:23,088 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:24:23,091 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:24:23,091 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:24:31,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.17 % 2021-12-02 05:24:31,982 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:24:31,982 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:24:33,101 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.17 % 2021-12-02 05:24:33,101 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 12.17 2021-12-02 05:24:33,115 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:24:33,118 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:24:33,118 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:24:41,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.17 % 2021-12-02 05:24:42,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:24:42,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:24:43,130 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.17 % 2021-12-02 05:24:43,130 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 12.17 2021-12-02 05:24:43,145 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:24:43,148 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:24:43,148 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:24:52,017 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.17 % 2021-12-02 05:24:52,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:24:52,025 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:24:53,160 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.17 % 2021-12-02 05:24:53,160 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 12.17 2021-12-02 05:24:53,177 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:24:53,178 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:24:53,180 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:25:02,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.17 % 2021-12-02 05:25:02,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:25:02,063 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:25:03,190 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.53 % 2021-12-02 05:25:03,190 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 12.53 2021-12-02 05:25:03,204 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:25:03,207 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:25:03,207 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:25:12,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.53 % 2021-12-02 05:25:12,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:25:12,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:25:13,219 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.53 % 2021-12-02 05:25:13,219 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 12.53 2021-12-02 05:25:13,233 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:25:13,236 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:25:13,236 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:25:22,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.53 % 2021-12-02 05:25:22,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:25:22,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:25:23,247 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.53 % 2021-12-02 05:25:23,247 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 12.53 2021-12-02 05:25:23,262 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:25:23,263 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:25:23,263 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:25:32,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.41 % 2021-12-02 05:25:32,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:25:32,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:25:33,276 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.41 % 2021-12-02 05:25:33,276 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 13.41 2021-12-02 05:25:33,296 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:25:33,299 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:25:33,299 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:25:42,130 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.41 % 2021-12-02 05:25:42,137 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:25:42,137 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:25:43,309 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.41 % 2021-12-02 05:25:43,309 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 13.41 2021-12-02 05:25:43,323 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:25:43,325 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:25:43,325 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:25:52,150 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.41 % 2021-12-02 05:25:52,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:25:52,163 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:25:53,336 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.41 % 2021-12-02 05:25:53,336 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 13.41 2021-12-02 05:25:53,349 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:25:53,351 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:25:53,351 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:26:02,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.83 % 2021-12-02 05:26:02,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:26:02,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:26:03,364 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.83 % 2021-12-02 05:26:03,364 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 13.83 2021-12-02 05:26:03,380 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:26:03,381 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:26:03,381 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:26:12,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.19 % 2021-12-02 05:26:12,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:26:12,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:26:13,394 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.19 % 2021-12-02 05:26:13,394 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 14.19 2021-12-02 05:26:13,414 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:26:13,417 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:26:13,417 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:26:22,209 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.19 % 2021-12-02 05:26:22,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:26:22,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:26:23,430 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.19 % 2021-12-02 05:26:23,430 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 14.19 2021-12-02 05:26:23,446 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:26:23,448 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:26:23,448 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:26:32,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.19 % 2021-12-02 05:26:32,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:26:32,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:26:33,461 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.19 % 2021-12-02 05:26:33,461 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 14.19 2021-12-02 05:26:33,483 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:26:33,484 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:26:33,486 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:26:42,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.19 % 2021-12-02 05:26:42,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:26:42,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:26:43,497 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.19 % 2021-12-02 05:26:43,497 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 14.19 2021-12-02 05:26:43,513 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:26:43,515 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:26:43,516 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:26:52,260 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.66 % 2021-12-02 05:26:52,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:26:52,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:26:53,526 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.66 % 2021-12-02 05:26:53,526 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 14.66 2021-12-02 05:26:53,540 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:26:53,542 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:26:53,543 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:27:02,278 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.66 % 2021-12-02 05:27:02,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:27:02,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:27:03,555 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.66 % 2021-12-02 05:27:03,555 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 14.66 2021-12-02 05:27:03,572 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:27:03,573 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:27:03,573 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:27:12,293 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.66 % 2021-12-02 05:27:12,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:27:12,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:27:13,585 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.66 % 2021-12-02 05:27:13,585 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 14.66 2021-12-02 05:27:13,601 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:27:13,605 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:27:13,605 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:27:22,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.02 % 2021-12-02 05:27:22,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:27:22,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:27:23,618 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.02 % 2021-12-02 05:27:23,618 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 15.02 2021-12-02 05:27:23,635 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:27:23,638 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:27:23,638 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:27:32,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.90 % 2021-12-02 05:27:32,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:27:32,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:27:33,661 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.90 % 2021-12-02 05:27:33,661 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 15.9 2021-12-02 05:27:33,683 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:27:33,686 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:27:33,686 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:27:42,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.90 % 2021-12-02 05:27:42,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:27:42,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:27:43,697 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.90 % 2021-12-02 05:27:43,697 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 15.9 2021-12-02 05:27:43,713 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:27:43,716 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:27:43,716 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:27:52,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.90 % 2021-12-02 05:27:52,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:27:52,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:27:53,726 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.90 % 2021-12-02 05:27:53,726 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 15.9 2021-12-02 05:27:53,743 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:27:53,746 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:27:53,747 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:28:02,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.90 % 2021-12-02 05:28:02,382 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:28:02,382 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:28:03,760 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.90 % 2021-12-02 05:28:03,760 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 15.9 2021-12-02 05:28:03,776 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:28:03,779 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:28:03,779 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:28:12,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.31 % 2021-12-02 05:28:12,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:28:12,400 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:28:13,790 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.31 % 2021-12-02 05:28:13,792 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 16.31 2021-12-02 05:28:13,806 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:28:13,809 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:28:13,809 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:28:22,413 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.31 % 2021-12-02 05:28:22,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:28:22,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:28:23,819 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.31 % 2021-12-02 05:28:23,819 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 16.31 2021-12-02 05:28:23,835 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:28:23,838 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:28:23,838 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:28:32,431 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.31 % 2021-12-02 05:28:32,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:28:32,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:28:33,851 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.31 % 2021-12-02 05:28:33,852 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 16.31 2021-12-02 05:28:33,871 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:28:33,875 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:28:33,875 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:28:42,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.31 % 2021-12-02 05:28:42,456 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:28:42,456 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:28:43,891 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.31 % 2021-12-02 05:28:43,891 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 16.31 2021-12-02 05:28:43,905 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:28:43,908 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:28:43,908 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:28:52,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.19 % 2021-12-02 05:28:52,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:28:52,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:28:53,921 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.19 % 2021-12-02 05:28:53,921 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 17.19 2021-12-02 05:28:53,936 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:28:53,938 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:28:53,938 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:29:02,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.19 % 2021-12-02 05:29:02,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:29:02,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:29:03,950 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.19 % 2021-12-02 05:29:03,950 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 17.19 2021-12-02 05:29:03,966 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:29:03,967 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:29:03,967 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:29:12,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.56 % 2021-12-02 05:29:12,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:29:12,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:29:13,979 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.56 % 2021-12-02 05:29:13,980 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 17.56 2021-12-02 05:29:13,996 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:29:13,999 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:29:13,999 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:29:22,517 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.56 % 2021-12-02 05:29:22,525 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:29:22,525 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:29:24,026 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.56 % 2021-12-02 05:29:24,026 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 17.56 2021-12-02 05:29:24,039 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:29:24,042 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:29:24,042 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:29:32,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.02 % 2021-12-02 05:29:32,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:29:32,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:29:34,053 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.02 % 2021-12-02 05:29:34,053 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 18.02 2021-12-02 05:29:34,072 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:29:34,075 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:29:34,075 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:29:42,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.02 % 2021-12-02 05:29:42,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:29:42,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:29:44,085 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.02 % 2021-12-02 05:29:44,085 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 18.02 2021-12-02 05:29:44,101 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:29:44,104 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:29:44,104 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:29:52,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.02 % 2021-12-02 05:29:52,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:29:52,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:29:54,115 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.02 % 2021-12-02 05:29:54,115 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 18.02 2021-12-02 05:29:54,131 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:29:54,134 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:29:54,135 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:30:02,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.02 % 2021-12-02 05:30:02,599 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:30:02,599 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:30:04,148 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.02 % 2021-12-02 05:30:04,148 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 18.02 2021-12-02 05:30:04,167 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:30:04,174 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:30:04,174 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:30:12,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.02 % 2021-12-02 05:30:12,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:30:12,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:30:14,186 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.02 % 2021-12-02 05:30:14,187 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 18.02 2021-12-02 05:30:14,203 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:30:14,207 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:30:14,207 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:30:22,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.39 % 2021-12-02 05:30:22,645 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:30:22,645 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:30:24,219 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.39 % 2021-12-02 05:30:24,220 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 18.39 2021-12-02 05:30:24,240 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:30:24,242 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:30:24,242 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:30:32,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.39 % 2021-12-02 05:30:32,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:30:32,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:30:34,253 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.26 % 2021-12-02 05:30:34,253 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 19.26 2021-12-02 05:30:34,269 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:30:34,270 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:30:34,270 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:30:42,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.26 % 2021-12-02 05:30:42,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:30:42,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:30:44,280 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.26 % 2021-12-02 05:30:44,280 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 19.26 2021-12-02 05:30:44,296 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:30:44,298 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:30:44,298 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:30:52,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.26 % 2021-12-02 05:30:52,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:30:52,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:30:54,308 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.26 % 2021-12-02 05:30:54,309 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 19.26 2021-12-02 05:30:54,322 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:30:54,325 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:30:54,325 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:31:02,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.26 % 2021-12-02 05:31:02,724 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:31:02,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:31:04,335 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.26 % 2021-12-02 05:31:04,335 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 19.26 2021-12-02 05:31:04,349 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:31:04,351 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:31:04,351 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:31:12,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.68 % 2021-12-02 05:31:12,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:31:12,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:31:14,364 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.68 % 2021-12-02 05:31:14,364 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 19.68 2021-12-02 05:31:14,380 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:31:14,381 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:31:14,381 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:31:22,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.68 % 2021-12-02 05:31:22,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:31:22,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:31:24,394 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.68 % 2021-12-02 05:31:24,394 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 19.68 2021-12-02 05:31:24,414 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:31:24,415 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:31:24,415 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:31:32,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.68 % 2021-12-02 05:31:32,780 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:31:32,780 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:31:34,427 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.68 % 2021-12-02 05:31:34,427 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 19.68 2021-12-02 05:31:34,443 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:31:34,444 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:31:34,444 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:31:42,790 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.04 % 2021-12-02 05:31:42,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:31:42,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:31:44,457 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.04 % 2021-12-02 05:31:44,457 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 20.04 2021-12-02 05:31:44,476 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:31:44,479 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:31:44,479 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:31:52,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.04 % 2021-12-02 05:31:52,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:31:52,823 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:31:54,490 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.04 % 2021-12-02 05:31:54,492 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 20.04 2021-12-02 05:31:54,513 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:31:54,516 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:31:54,516 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:32:02,836 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.51 % 2021-12-02 05:32:02,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:32:02,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:32:04,529 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.51 % 2021-12-02 05:32:04,529 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 20.51 2021-12-02 05:32:04,549 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:32:04,551 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:32:04,552 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:32:12,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.97 % 2021-12-02 05:32:12,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:32:12,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:32:14,562 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.97 % 2021-12-02 05:32:14,562 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 20.97 2021-12-02 05:32:14,575 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:32:14,576 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:32:14,578 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:32:22,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.97 % 2021-12-02 05:32:22,887 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:32:22,887 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:32:24,589 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.97 % 2021-12-02 05:32:24,591 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 20.97 2021-12-02 05:32:24,608 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:32:24,609 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:32:24,611 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:32:32,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.97 % 2021-12-02 05:32:32,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:32:32,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:32:34,621 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.34 % 2021-12-02 05:32:34,621 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 21.34 2021-12-02 05:32:34,638 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:32:34,640 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:32:34,641 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:32:42,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.34 % 2021-12-02 05:32:42,923 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:32:42,923 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:32:44,651 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.34 % 2021-12-02 05:32:44,653 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 21.34 2021-12-02 05:32:44,671 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:32:44,674 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:32:44,674 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:32:52,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.34 % 2021-12-02 05:32:52,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:32:52,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:32:54,686 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.34 % 2021-12-02 05:32:54,686 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 21.34 2021-12-02 05:32:54,703 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:32:54,706 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:32:54,706 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:33:02,951 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.34 % 2021-12-02 05:33:02,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:33:02,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:33:04,719 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.34 % 2021-12-02 05:33:04,719 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 21.34 2021-12-02 05:33:04,737 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:33:04,740 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:33:04,740 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:33:12,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.75 % 2021-12-02 05:33:12,977 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:33:12,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:33:14,752 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.75 % 2021-12-02 05:33:14,752 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 21.75 2021-12-02 05:33:14,766 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:33:14,769 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:33:14,769 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:33:22,989 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.75 % 2021-12-02 05:33:22,997 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:33:22,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:33:24,782 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.75 % 2021-12-02 05:33:24,782 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 21.75 2021-12-02 05:33:24,799 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:33:24,802 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:33:24,802 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:33:33,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.75 % 2021-12-02 05:33:33,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:33:33,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:33:34,815 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.75 % 2021-12-02 05:33:34,815 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 21.75 2021-12-02 05:33:34,831 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:33:34,832 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:33:34,832 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:33:43,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.63 % 2021-12-02 05:33:43,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:33:43,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:33:44,845 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.63 % 2021-12-02 05:33:44,845 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 22.63 2021-12-02 05:33:44,861 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:33:44,865 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:33:44,865 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:33:53,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.63 % 2021-12-02 05:33:53,073 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:33:53,073 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:33:54,875 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.63 % 2021-12-02 05:33:54,875 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 22.63 2021-12-02 05:33:54,891 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:33:54,894 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:33:54,894 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:34:03,084 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 22.63 % 2021-12-02 05:34:03,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:34:03,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:34:04,923 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.00 % 2021-12-02 05:34:04,923 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 23.0 2021-12-02 05:34:04,937 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:34:04,940 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:34:04,940 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:34:13,104 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.00 % 2021-12-02 05:34:13,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:34:13,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:34:14,951 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.00 % 2021-12-02 05:34:14,953 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 23.0 2021-12-02 05:34:14,970 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:34:14,973 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:34:14,973 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:34:23,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.00 % 2021-12-02 05:34:23,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:34:23,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:34:24,983 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.00 % 2021-12-02 05:34:24,983 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 23.0 2021-12-02 05:34:24,997 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:34:25,000 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:34:25,000 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:34:33,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.46 % 2021-12-02 05:34:33,150 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:34:33,150 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:34:35,010 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.46 % 2021-12-02 05:34:35,010 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 23.46 2021-12-02 05:34:35,026 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:34:35,029 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:34:35,029 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:34:43,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.46 % 2021-12-02 05:34:43,168 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:34:43,170 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:34:45,040 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.46 % 2021-12-02 05:34:45,040 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 23.46 2021-12-02 05:34:45,058 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:34:45,061 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:34:45,061 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:34:53,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.46 % 2021-12-02 05:34:53,186 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:34:53,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:34:55,073 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.46 % 2021-12-02 05:34:55,073 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 23.46 2021-12-02 05:34:55,092 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:34:55,096 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:34:55,096 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:35:03,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.46 % 2021-12-02 05:35:03,204 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:35:03,204 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:35:05,108 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.46 % 2021-12-02 05:35:05,108 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 23.46 2021-12-02 05:35:05,124 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:35:05,127 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:35:05,127 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:35:13,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.87 % 2021-12-02 05:35:13,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:35:13,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:35:15,138 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.87 % 2021-12-02 05:35:15,140 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 23.87 2021-12-02 05:35:15,155 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:35:15,157 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:35:15,157 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:35:23,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.65 % 2021-12-02 05:35:23,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:35:23,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:35:25,170 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.65 % 2021-12-02 05:35:25,170 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 24.65 2021-12-02 05:35:25,183 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:35:25,184 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:35:25,186 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:35:33,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.65 % 2021-12-02 05:35:33,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:35:33,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:35:35,198 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.65 % 2021-12-02 05:35:35,198 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 24.65 2021-12-02 05:35:35,213 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:35:35,217 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:35:35,217 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:35:43,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.65 % 2021-12-02 05:35:43,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:35:43,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:35:45,230 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.65 % 2021-12-02 05:35:45,230 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 24.65 2021-12-02 05:35:45,246 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:35:45,247 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:35:45,249 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:35:53,293 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.65 % 2021-12-02 05:35:53,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:35:53,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:35:55,259 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.65 % 2021-12-02 05:35:55,260 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 24.65 2021-12-02 05:35:55,276 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:35:55,278 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:35:55,279 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:36:03,313 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.65 % 2021-12-02 05:36:03,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:36:03,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:36:05,308 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.65 % 2021-12-02 05:36:05,309 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 24.65 2021-12-02 05:36:05,325 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:36:05,328 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:36:05,328 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:36:13,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.65 % 2021-12-02 05:36:13,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:36:13,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:36:15,342 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 24.65 % 2021-12-02 05:36:15,342 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 24.65 2021-12-02 05:36:15,358 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:36:15,359 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:36:15,359 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:36:23,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.12 % 2021-12-02 05:36:23,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:36:23,357 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:36:25,371 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.12 % 2021-12-02 05:36:25,371 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 25.12 2021-12-02 05:36:25,387 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:36:25,388 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:36:25,390 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:36:33,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.58 % 2021-12-02 05:36:33,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:36:33,374 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:36:35,401 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.58 % 2021-12-02 05:36:35,401 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 25.58 2021-12-02 05:36:35,415 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:36:35,418 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:36:35,418 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:36:43,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.58 % 2021-12-02 05:36:43,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:36:43,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:36:45,430 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.58 % 2021-12-02 05:36:45,430 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 25.58 2021-12-02 05:36:45,447 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:36:45,450 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:36:45,450 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:36:53,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.58 % 2021-12-02 05:36:53,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:36:53,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:36:55,460 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.58 % 2021-12-02 05:36:55,460 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 25.58 2021-12-02 05:36:55,476 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:36:55,477 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:36:55,477 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:37:03,418 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.58 % 2021-12-02 05:37:03,426 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:37:03,426 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:37:05,490 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.58 % 2021-12-02 05:37:05,490 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 25.58 2021-12-02 05:37:05,509 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:37:05,510 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:37:05,510 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:37:13,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.31 % 2021-12-02 05:37:13,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:37:13,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:37:15,522 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.31 % 2021-12-02 05:37:15,523 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 26.31 2021-12-02 05:37:15,538 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:37:15,539 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:37:15,539 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:37:23,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.31 % 2021-12-02 05:37:23,466 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:37:23,466 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:37:25,552 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.31 % 2021-12-02 05:37:25,552 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 26.31 2021-12-02 05:37:25,568 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:37:25,569 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:37:25,569 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:37:33,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.31 % 2021-12-02 05:37:33,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:37:33,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:37:35,581 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.31 % 2021-12-02 05:37:35,581 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 26.31 2021-12-02 05:37:35,594 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:37:35,595 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:37:35,595 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:37:43,492 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.31 % 2021-12-02 05:37:43,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:37:43,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:37:45,605 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.31 % 2021-12-02 05:37:45,607 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 26.31 2021-12-02 05:37:45,621 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:37:45,625 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:37:45,625 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:37:53,507 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.78 % 2021-12-02 05:37:53,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:37:53,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:37:55,634 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 26.78 % 2021-12-02 05:37:55,635 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 26.78 2021-12-02 05:37:55,653 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:37:55,654 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:37:55,654 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:38:03,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.19 % 2021-12-02 05:38:03,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:38:03,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:38:05,667 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.19 % 2021-12-02 05:38:05,667 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 27.19 2021-12-02 05:38:05,687 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:38:05,690 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:38:05,690 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:38:13,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.19 % 2021-12-02 05:38:13,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:38:13,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:38:15,703 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.19 % 2021-12-02 05:38:15,703 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 27.19 2021-12-02 05:38:15,719 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:38:15,720 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:38:15,721 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:38:23,558 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.19 % 2021-12-02 05:38:23,563 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:38:23,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:38:25,732 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.19 % 2021-12-02 05:38:25,732 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 27.19 2021-12-02 05:38:25,746 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:38:25,749 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:38:25,749 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:38:33,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:38:33,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:38:33,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:38:35,759 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:38:35,759 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 28.02 2021-12-02 05:38:35,775 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:38:35,778 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:38:35,778 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:38:43,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:38:43,596 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:38:43,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:38:45,788 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:38:45,788 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 28.02 2021-12-02 05:38:45,802 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:38:45,803 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:38:45,805 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:38:53,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:38:53,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:38:53,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:38:55,818 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:38:55,818 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 28.02 2021-12-02 05:38:55,832 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:38:55,835 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:38:55,835 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:39:03,627 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:39:03,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:39:03,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:39:05,846 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:39:05,846 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 28.02 2021-12-02 05:39:05,862 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:39:05,865 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:39:05,865 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:39:13,647 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:39:13,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:39:13,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:39:15,875 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:39:15,875 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 28.02 2021-12-02 05:39:15,891 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:39:15,894 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:39:15,894 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:39:23,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:39:23,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:39:23,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:39:25,905 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:39:25,905 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 28.02 2021-12-02 05:39:25,923 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:39:25,924 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:39:25,924 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:39:33,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.02 % 2021-12-02 05:39:33,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:39:33,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:39:35,937 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.43 % 2021-12-02 05:39:35,937 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 28.43 2021-12-02 05:39:35,956 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:39:35,959 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:39:35,959 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:39:43,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.43 % 2021-12-02 05:39:43,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:39:43,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:39:45,969 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.43 % 2021-12-02 05:39:45,970 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 28.43 2021-12-02 05:39:45,986 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:39:45,989 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:39:45,989 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:39:53,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.26 % 2021-12-02 05:39:53,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:39:53,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:39:56,000 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.26 % 2021-12-02 05:39:56,000 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 29.26 2021-12-02 05:39:56,017 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:39:56,019 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:39:56,019 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:40:03,752 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.26 % 2021-12-02 05:40:03,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:40:03,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:40:06,032 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.26 % 2021-12-02 05:40:06,032 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 29.26 2021-12-02 05:40:06,048 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:40:06,051 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:40:06,051 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:40:13,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.68 % 2021-12-02 05:40:13,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:40:13,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:40:16,062 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.68 % 2021-12-02 05:40:16,062 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 29.68 2021-12-02 05:40:16,079 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:40:16,082 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:40:16,084 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:40:23,803 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.68 % 2021-12-02 05:40:23,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:40:23,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:40:26,098 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.68 % 2021-12-02 05:40:26,098 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 29.68 2021-12-02 05:40:26,115 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:40:26,119 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:40:26,119 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:40:33,826 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.68 % 2021-12-02 05:40:33,836 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:40:33,836 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:40:36,134 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.68 % 2021-12-02 05:40:36,134 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 29.68 2021-12-02 05:40:36,150 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:40:36,153 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:40:36,153 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:40:43,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.68 % 2021-12-02 05:40:43,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:40:43,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:40:46,164 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.68 % 2021-12-02 05:40:46,164 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 29.68 2021-12-02 05:40:46,178 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:40:46,181 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:40:46,181 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:40:53,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.68 % 2021-12-02 05:40:53,880 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:40:53,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:40:56,191 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.68 % 2021-12-02 05:40:56,193 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 29.68 2021-12-02 05:40:56,207 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:40:56,210 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:40:56,210 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:41:03,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.68 % 2021-12-02 05:41:03,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:41:03,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:41:06,220 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.14 % 2021-12-02 05:41:06,220 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 30.14 2021-12-02 05:41:06,234 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:41:06,237 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:41:06,237 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:41:13,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.56 % 2021-12-02 05:41:13,923 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:41:13,923 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:41:16,249 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.56 % 2021-12-02 05:41:16,249 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 30.56 2021-12-02 05:41:16,265 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:41:16,267 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:41:16,267 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:41:23,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.56 % 2021-12-02 05:41:23,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:41:23,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:41:26,279 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.56 % 2021-12-02 05:41:26,280 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 30.56 2021-12-02 05:41:26,312 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:41:26,315 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:41:26,315 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:41:33,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.92 % 2021-12-02 05:41:33,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:41:33,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:41:36,326 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 30.92 % 2021-12-02 05:41:36,326 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 30.92 2021-12-02 05:41:36,344 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:41:36,346 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:41:36,346 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:41:43,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:41:43,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:41:43,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:41:46,358 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:41:46,358 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 31.38 2021-12-02 05:41:46,372 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:41:46,375 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:41:46,375 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:41:54,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:41:54,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:41:54,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:41:56,385 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:41:56,385 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 31.38 2021-12-02 05:41:56,400 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:41:56,401 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:41:56,401 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:42:04,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:42:04,030 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:42:04,030 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:42:06,413 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:42:06,414 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 31.38 2021-12-02 05:42:06,427 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:42:06,430 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:42:06,430 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:42:14,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:42:14,051 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:42:14,051 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:42:16,440 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:42:16,440 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 31.38 2021-12-02 05:42:16,457 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:42:16,459 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:42:16,459 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:42:24,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:42:24,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:42:24,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:42:26,469 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:42:26,470 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 31.38 2021-12-02 05:42:26,486 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:42:26,487 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:42:26,489 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:42:34,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:42:34,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:42:34,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:42:36,499 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:42:36,499 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 31.38 2021-12-02 05:42:36,513 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:42:36,516 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:42:36,516 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:42:44,102 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:42:44,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:42:44,112 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:42:46,529 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.38 % 2021-12-02 05:42:46,529 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 31.38 2021-12-02 05:42:46,549 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:42:46,551 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:42:46,551 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:42:54,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.80 % 2021-12-02 05:42:54,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:42:54,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:42:56,562 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.63 % 2021-12-02 05:42:56,562 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 32.63 2021-12-02 05:42:56,578 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:42:56,581 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:42:56,581 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:43:04,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:43:04,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:43:04,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:43:06,592 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:43:06,592 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 33.04 2021-12-02 05:43:06,615 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:43:06,617 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:43:06,617 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:43:14,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:43:14,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:43:14,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:43:16,628 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:43:16,630 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 33.04 2021-12-02 05:43:16,648 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:43:16,651 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:43:16,651 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:43:24,186 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:43:24,196 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:43:24,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:43:26,663 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:43:26,664 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 33.04 2021-12-02 05:43:26,680 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:43:26,684 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:43:26,684 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:43:34,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:43:34,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:43:34,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:43:36,694 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:43:36,694 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 33.04 2021-12-02 05:43:36,714 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:43:36,717 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:43:36,717 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:43:44,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:43:44,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:43:44,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:43:46,729 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:43:46,729 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 33.04 2021-12-02 05:43:46,747 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:43:46,752 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:43:46,752 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:43:54,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:43:54,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:43:54,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:43:56,763 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:43:56,763 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 33.04 2021-12-02 05:43:56,779 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:43:56,782 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:43:56,782 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:44:04,278 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.04 % 2021-12-02 05:44:04,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:44:04,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:44:06,792 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.41 % 2021-12-02 05:44:06,792 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 33.41 2021-12-02 05:44:06,808 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:44:06,809 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:44:06,809 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:44:14,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.41 % 2021-12-02 05:44:14,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:44:14,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:44:16,822 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.41 % 2021-12-02 05:44:16,822 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 33.41 2021-12-02 05:44:16,836 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:44:16,838 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:44:16,838 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:44:24,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.87 % 2021-12-02 05:44:24,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:44:24,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:44:26,851 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.87 % 2021-12-02 05:44:26,851 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 33.87 2021-12-02 05:44:26,865 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:44:26,868 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:44:26,868 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:44:34,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.87 % 2021-12-02 05:44:34,339 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:44:34,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:44:36,881 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 33.87 % 2021-12-02 05:44:36,881 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 33.87 2021-12-02 05:44:36,900 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:44:36,903 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:44:36,903 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:44:44,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.75 % 2021-12-02 05:44:44,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:44:44,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:44:46,914 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.75 % 2021-12-02 05:44:46,914 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 34.75 2021-12-02 05:44:46,931 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:44:46,934 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:44:46,934 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:44:54,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.75 % 2021-12-02 05:44:54,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:44:54,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:44:56,946 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.75 % 2021-12-02 05:44:56,946 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 34.75 2021-12-02 05:44:56,960 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:44:56,963 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:44:56,963 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:45:04,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.75 % 2021-12-02 05:45:04,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:45:04,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:45:06,973 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.75 % 2021-12-02 05:45:06,973 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 34.75 2021-12-02 05:45:06,989 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:45:06,990 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:45:06,990 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:45:14,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.75 % 2021-12-02 05:45:14,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:45:14,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:45:17,002 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.75 % 2021-12-02 05:45:17,002 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 34.75 2021-12-02 05:45:17,016 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:45:17,019 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:45:17,019 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:45:24,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.75 % 2021-12-02 05:45:24,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:45:24,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:45:27,029 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.75 % 2021-12-02 05:45:27,029 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 34.75 2021-12-02 05:45:27,043 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:45:27,048 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:45:27,048 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:45:34,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.75 % 2021-12-02 05:45:34,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:45:34,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:45:37,062 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.12 % 2021-12-02 05:45:37,062 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 35.12 2021-12-02 05:45:37,076 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:45:37,079 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:45:37,079 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:45:44,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.12 % 2021-12-02 05:45:44,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:45:44,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:45:47,092 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.12 % 2021-12-02 05:45:47,092 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 35.12 2021-12-02 05:45:47,108 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:45:47,111 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:45:47,111 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:45:54,471 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.12 % 2021-12-02 05:45:54,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:45:54,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:45:57,121 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.12 % 2021-12-02 05:45:57,122 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 35.12 2021-12-02 05:45:57,138 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:45:57,140 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:45:57,140 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:46:04,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.53 % 2021-12-02 05:46:04,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:46:04,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:46:07,151 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.53 % 2021-12-02 05:46:07,151 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 35.53 2021-12-02 05:46:07,170 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:46:07,173 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:46:07,174 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:46:14,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.00 % 2021-12-02 05:46:14,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:46:14,512 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:46:17,184 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.00 % 2021-12-02 05:46:17,184 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 36.0 2021-12-02 05:46:17,198 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:46:17,201 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:46:17,201 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:46:24,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.00 % 2021-12-02 05:46:24,528 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:46:24,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:46:27,211 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.41 % 2021-12-02 05:46:27,213 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 36.41 2021-12-02 05:46:27,227 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:46:27,230 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:46:27,230 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:46:34,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.41 % 2021-12-02 05:46:34,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:46:34,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:46:37,240 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.41 % 2021-12-02 05:46:37,242 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 36.41 2021-12-02 05:46:37,256 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:46:37,259 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:46:37,259 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:46:44,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.41 % 2021-12-02 05:46:44,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:46:44,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:46:47,272 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.41 % 2021-12-02 05:46:47,272 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 36.41 2021-12-02 05:46:47,289 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:46:47,292 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:46:47,292 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:46:54,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.41 % 2021-12-02 05:46:54,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:46:54,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:46:57,305 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.77 % 2021-12-02 05:46:57,305 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 36.77 2021-12-02 05:46:57,319 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:46:57,322 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:46:57,322 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:47:04,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.77 % 2021-12-02 05:47:04,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:47:04,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:47:07,335 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.77 % 2021-12-02 05:47:07,335 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 36.77 2021-12-02 05:47:07,349 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:47:07,351 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:47:07,352 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:47:14,602 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.77 % 2021-12-02 05:47:14,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:47:14,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:47:17,364 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.77 % 2021-12-02 05:47:17,365 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 36.77 2021-12-02 05:47:17,381 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:47:17,384 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:47:17,384 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:47:24,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.77 % 2021-12-02 05:47:24,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:47:24,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:47:27,392 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.77 % 2021-12-02 05:47:27,394 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 36.77 2021-12-02 05:47:27,407 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:47:27,410 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:47:27,410 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:47:34,632 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.77 % 2021-12-02 05:47:34,640 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:47:34,640 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:47:37,420 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 36.77 % 2021-12-02 05:47:37,420 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 36.77 2021-12-02 05:47:37,434 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:47:37,436 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:47:37,436 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:47:44,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.24 % 2021-12-02 05:47:44,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:47:44,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:47:47,447 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.24 % 2021-12-02 05:47:47,447 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 37.24 2021-12-02 05:47:47,463 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:47:47,464 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:47:47,464 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:47:54,665 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.24 % 2021-12-02 05:47:54,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:47:54,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:47:57,476 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.24 % 2021-12-02 05:47:57,476 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 37.24 2021-12-02 05:47:57,492 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:47:57,493 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:47:57,493 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:48:04,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.65 % 2021-12-02 05:48:04,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:48:04,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:48:07,503 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.65 % 2021-12-02 05:48:07,503 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 37.65 2021-12-02 05:48:07,519 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:48:07,522 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:48:07,522 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:48:14,714 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.48 % 2021-12-02 05:48:14,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:48:14,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:48:17,532 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.48 % 2021-12-02 05:48:17,533 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 38.48 2021-12-02 05:48:17,553 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:48:17,555 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:48:17,555 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:48:24,734 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.48 % 2021-12-02 05:48:24,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:48:24,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:48:27,566 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.48 % 2021-12-02 05:48:27,568 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 38.48 2021-12-02 05:48:27,585 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:48:27,588 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:48:27,588 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:48:34,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.48 % 2021-12-02 05:48:34,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:48:34,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:48:37,598 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.48 % 2021-12-02 05:48:37,598 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 38.48 2021-12-02 05:48:37,611 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:48:37,614 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:48:37,614 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:48:44,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.48 % 2021-12-02 05:48:44,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:48:44,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:48:47,624 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.48 % 2021-12-02 05:48:47,625 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 38.48 2021-12-02 05:48:47,638 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:48:47,640 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:48:47,640 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:48:54,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.48 % 2021-12-02 05:48:54,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:48:54,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:48:57,651 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.48 % 2021-12-02 05:48:57,651 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 38.48 2021-12-02 05:48:57,664 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:48:57,665 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:48:57,667 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:49:04,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.48 % 2021-12-02 05:49:04,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:49:04,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:49:07,677 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.48 % 2021-12-02 05:49:07,678 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 38.48 2021-12-02 05:49:07,693 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:49:07,696 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:49:07,696 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:49:14,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.90 % 2021-12-02 05:49:14,844 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:49:14,844 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:49:17,710 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.90 % 2021-12-02 05:49:17,710 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 38.9 2021-12-02 05:49:17,726 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:49:17,729 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:49:17,729 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:49:24,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.90 % 2021-12-02 05:49:24,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:49:24,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:49:27,743 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.90 % 2021-12-02 05:49:27,743 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 38.9 2021-12-02 05:49:27,763 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:49:27,766 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:49:27,766 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:49:34,882 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.90 % 2021-12-02 05:49:34,890 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:49:34,890 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:49:37,779 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.26 % 2021-12-02 05:49:37,779 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 39.26 2021-12-02 05:49:37,796 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:49:37,799 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:49:37,799 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:49:44,903 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.73 % 2021-12-02 05:49:44,908 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:49:44,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:49:47,812 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.73 % 2021-12-02 05:49:47,813 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 39.73 2021-12-02 05:49:47,828 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:49:47,831 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:49:47,831 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:49:54,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.14 % 2021-12-02 05:49:54,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:49:54,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:49:57,845 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.14 % 2021-12-02 05:49:57,845 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 40.14 2021-12-02 05:49:57,859 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:49:57,861 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:49:57,862 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:50:04,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.14 % 2021-12-02 05:50:04,956 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:50:04,956 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:50:07,875 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.14 % 2021-12-02 05:50:07,875 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 40.14 2021-12-02 05:50:07,891 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:50:07,892 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:50:07,894 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:50:14,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.14 % 2021-12-02 05:50:14,982 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:50:14,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:50:17,907 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.14 % 2021-12-02 05:50:17,907 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 40.14 2021-12-02 05:50:17,921 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:50:17,924 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:50:17,924 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:50:24,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.14 % 2021-12-02 05:50:25,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:50:25,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:50:27,940 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.14 % 2021-12-02 05:50:27,940 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 40.14 2021-12-02 05:50:27,956 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:50:27,957 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:50:27,957 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:50:35,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.14 % 2021-12-02 05:50:35,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:50:35,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:50:37,976 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.61 % 2021-12-02 05:50:37,976 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 40.61 2021-12-02 05:50:37,993 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:50:37,996 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:50:37,996 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:50:45,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.61 % 2021-12-02 05:50:45,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:50:45,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:50:48,012 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.61 % 2021-12-02 05:50:48,013 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 40.61 2021-12-02 05:50:48,028 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:50:48,029 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:50:48,030 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:50:55,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.61 % 2021-12-02 05:50:55,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:50:55,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:50:58,043 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.61 % 2021-12-02 05:50:58,043 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 40.61 2021-12-02 05:50:58,058 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:50:58,061 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:50:58,061 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:51:05,086 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.97 % 2021-12-02 05:51:05,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:51:05,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:51:08,088 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.97 % 2021-12-02 05:51:08,088 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 40.97 2021-12-02 05:51:08,102 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:51:08,105 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:51:08,105 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:51:15,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 41.38 % 2021-12-02 05:51:15,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:51:15,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:51:18,119 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 41.38 % 2021-12-02 05:51:18,121 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 41.38 2021-12-02 05:51:18,135 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:51:18,138 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:51:18,138 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:51:25,132 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 41.38 % 2021-12-02 05:51:25,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:51:25,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:51:28,164 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 41.38 % 2021-12-02 05:51:28,164 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 41.38 2021-12-02 05:51:28,178 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:51:28,181 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:51:28,181 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:51:35,158 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 41.85 % 2021-12-02 05:51:35,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:51:35,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:51:38,194 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 41.85 % 2021-12-02 05:51:38,194 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 41.85 2021-12-02 05:51:38,210 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:51:38,211 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:51:38,211 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:51:45,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 41.85 % 2021-12-02 05:51:45,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:51:45,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:51:48,224 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 41.85 % 2021-12-02 05:51:48,224 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 41.85 2021-12-02 05:51:48,240 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:51:48,243 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:51:48,243 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:51:55,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 41.85 % 2021-12-02 05:51:55,206 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:51:55,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:51:58,257 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 41.85 % 2021-12-02 05:51:58,257 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 41.85 2021-12-02 05:51:58,275 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:51:58,278 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:51:58,278 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:52:05,219 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.26 % 2021-12-02 05:52:05,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:52:05,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:52:08,292 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.26 % 2021-12-02 05:52:08,292 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 42.26 2021-12-02 05:52:08,308 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:52:08,311 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:52:08,311 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:52:15,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.26 % 2021-12-02 05:52:15,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:52:15,247 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:52:18,323 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.26 % 2021-12-02 05:52:18,323 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 42.26 2021-12-02 05:52:18,338 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:52:18,341 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:52:18,341 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:52:25,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.63 % 2021-12-02 05:52:25,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:52:25,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:52:28,354 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.63 % 2021-12-02 05:52:28,354 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 42.63 2021-12-02 05:52:28,368 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:52:28,369 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:52:28,369 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:52:35,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.63 % 2021-12-02 05:52:35,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:52:35,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:52:38,382 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.63 % 2021-12-02 05:52:38,382 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 42.63 2021-12-02 05:52:38,398 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:52:38,401 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:52:38,401 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:52:45,323 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.63 % 2021-12-02 05:52:45,331 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:52:45,331 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:52:48,414 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.09 % 2021-12-02 05:52:48,414 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 43.09 2021-12-02 05:52:48,427 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:52:48,430 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:52:48,430 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:52:55,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.09 % 2021-12-02 05:52:55,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:52:55,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:52:58,444 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.09 % 2021-12-02 05:52:58,444 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 43.09 2021-12-02 05:52:58,457 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:52:58,460 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:52:58,460 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:53:05,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.09 % 2021-12-02 05:53:05,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:53:05,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:53:08,474 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.51 % 2021-12-02 05:53:08,474 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 43.51 2021-12-02 05:53:08,494 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:53:08,497 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:53:08,497 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:53:15,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.51 % 2021-12-02 05:53:15,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:53:15,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:53:18,513 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.51 % 2021-12-02 05:53:18,513 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 43.51 2021-12-02 05:53:18,529 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:53:18,532 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:53:18,532 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:53:25,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.51 % 2021-12-02 05:53:25,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:53:25,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:53:28,546 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.51 % 2021-12-02 05:53:28,546 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 43.51 2021-12-02 05:53:28,562 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:53:28,563 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:53:28,565 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:53:35,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.51 % 2021-12-02 05:53:35,426 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:53:35,426 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:53:38,578 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.51 % 2021-12-02 05:53:38,578 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 43.51 2021-12-02 05:53:38,595 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:53:38,598 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:53:38,598 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:53:45,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.51 % 2021-12-02 05:53:45,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:53:45,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:53:48,612 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.51 % 2021-12-02 05:53:48,612 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 43.51 2021-12-02 05:53:48,630 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:53:48,631 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:53:48,631 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:53:55,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.51 % 2021-12-02 05:53:55,466 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:53:55,466 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:53:58,644 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.97 % 2021-12-02 05:53:58,645 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 43.97 2021-12-02 05:53:58,660 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:53:58,661 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:53:58,661 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:54:05,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.97 % 2021-12-02 05:54:05,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:54:05,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:54:08,674 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 44.33 % 2021-12-02 05:54:08,674 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 44.33 2021-12-02 05:54:08,691 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:54:08,693 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:54:08,694 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:54:15,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 44.75 % 2021-12-02 05:54:15,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:54:15,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:54:18,706 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 44.75 % 2021-12-02 05:54:18,707 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 44.75 2021-12-02 05:54:18,720 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:54:18,723 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:54:18,723 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:54:25,528 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:54:25,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:54:25,538 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:54:28,740 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:54:28,740 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 45.22 2021-12-02 05:54:28,769 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:54:28,770 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:54:28,772 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:54:35,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:54:35,561 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:54:35,561 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:54:38,785 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:54:38,785 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 45.22 2021-12-02 05:54:38,801 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:54:38,803 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:54:38,803 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:54:45,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:54:45,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:54:45,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:54:48,819 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:54:48,819 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 45.22 2021-12-02 05:54:48,838 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:54:48,839 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:54:48,841 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:54:55,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:54:55,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:54:55,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:54:58,855 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:54:58,855 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 45.22 2021-12-02 05:54:58,877 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:54:58,878 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:54:58,880 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:55:05,624 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:55:05,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:55:05,630 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:55:08,894 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:55:08,895 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 45.22 2021-12-02 05:55:08,911 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:55:08,914 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:55:08,914 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:55:15,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:55:15,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:55:15,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:55:18,927 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:55:18,928 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 45.22 2021-12-02 05:55:18,947 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:55:18,948 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:55:18,948 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:55:25,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.22 % 2021-12-02 05:55:25,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:55:25,668 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:55:28,961 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.63 % 2021-12-02 05:55:28,961 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 45.63 2021-12-02 05:55:28,976 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:55:28,980 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:55:28,980 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:55:35,681 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.63 % 2021-12-02 05:55:35,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:55:35,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:55:38,994 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.63 % 2021-12-02 05:55:38,994 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 45.63 2021-12-02 05:55:39,012 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:55:39,015 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:55:39,016 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:55:45,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.99 % 2021-12-02 05:55:45,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:55:45,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:55:49,028 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.46 % 2021-12-02 05:55:49,029 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 46.46 2021-12-02 05:55:49,042 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:55:49,045 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:55:49,045 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:55:55,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.46 % 2021-12-02 05:55:55,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:55:55,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:55:59,059 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.46 % 2021-12-02 05:55:59,059 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 46.46 2021-12-02 05:55:59,078 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:55:59,082 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:55:59,082 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:56:05,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.87 % 2021-12-02 05:56:05,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:56:05,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:56:09,095 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.87 % 2021-12-02 05:56:09,095 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 46.87 2021-12-02 05:56:09,111 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:56:09,115 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:56:09,115 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:56:15,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.87 % 2021-12-02 05:56:15,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:56:15,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:56:19,130 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.87 % 2021-12-02 05:56:19,130 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 46.87 2021-12-02 05:56:19,148 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:56:19,151 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:56:19,151 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:56:25,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.87 % 2021-12-02 05:56:25,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:56:25,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:56:29,165 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.87 % 2021-12-02 05:56:29,165 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 46.87 2021-12-02 05:56:29,180 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:56:29,184 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:56:29,184 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:56:35,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.87 % 2021-12-02 05:56:35,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:56:35,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:56:39,198 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.87 % 2021-12-02 05:56:39,200 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 46.87 2021-12-02 05:56:39,216 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:56:39,219 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:56:39,220 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:56:45,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.87 % 2021-12-02 05:56:45,852 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:56:45,852 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:56:49,233 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.87 % 2021-12-02 05:56:49,233 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 46.87 2021-12-02 05:56:49,250 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:56:49,252 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:56:49,252 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:56:55,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.87 % 2021-12-02 05:56:55,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:56:55,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:56:59,265 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.87 % 2021-12-02 05:56:59,265 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 46.87 2021-12-02 05:56:59,282 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:56:59,283 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:56:59,283 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:57:05,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 47.34 % 2021-12-02 05:57:05,897 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:57:05,897 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:57:09,306 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 47.34 % 2021-12-02 05:57:09,306 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 47.34 2021-12-02 05:57:09,323 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:57:09,326 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:57:09,326 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:57:15,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 47.70 % 2021-12-02 05:57:15,918 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:57:15,918 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:57:19,338 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 47.70 % 2021-12-02 05:57:19,338 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 47.7 2021-12-02 05:57:19,355 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:57:19,358 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:57:19,358 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:57:25,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.11 % 2021-12-02 05:57:25,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:57:25,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:57:29,368 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.11 % 2021-12-02 05:57:29,369 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 48.11 2021-12-02 05:57:29,385 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:57:29,388 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:57:29,388 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:57:35,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.11 % 2021-12-02 05:57:35,961 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:57:35,961 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:57:39,400 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.58 % 2021-12-02 05:57:39,401 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 48.58 2021-12-02 05:57:39,424 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:57:39,426 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:57:39,426 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:57:45,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.58 % 2021-12-02 05:57:45,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:57:45,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:57:49,437 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.58 % 2021-12-02 05:57:49,437 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 48.58 2021-12-02 05:57:49,450 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:57:49,453 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:57:49,453 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:57:55,992 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.58 % 2021-12-02 05:57:56,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:57:56,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:57:59,464 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.58 % 2021-12-02 05:57:59,464 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 48.58 2021-12-02 05:57:59,480 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:57:59,482 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:57:59,482 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:58:06,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.58 % 2021-12-02 05:58:06,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:58:06,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:58:09,493 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.58 % 2021-12-02 05:58:09,493 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 48.58 2021-12-02 05:58:09,509 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:58:09,513 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:58:09,513 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:58:16,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.58 % 2021-12-02 05:58:16,048 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:58:16,048 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:58:19,526 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.58 % 2021-12-02 05:58:19,526 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 48.58 2021-12-02 05:58:19,543 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:58:19,546 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:58:19,546 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:58:26,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.58 % 2021-12-02 05:58:26,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:58:26,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:58:29,559 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.58 % 2021-12-02 05:58:29,559 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 48.58 2021-12-02 05:58:29,578 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:58:29,581 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:58:29,582 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:58:36,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.58 % 2021-12-02 05:58:36,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:58:36,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:58:39,594 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.94 % 2021-12-02 05:58:39,594 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 48.94 2021-12-02 05:58:39,609 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:58:39,612 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:58:39,612 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:58:46,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.94 % 2021-12-02 05:58:46,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:58:46,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:58:49,625 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 49.31 % 2021-12-02 05:58:49,625 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 49.31 2021-12-02 05:58:49,638 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:58:49,641 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:58:49,641 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:58:56,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 49.31 % 2021-12-02 05:58:56,137 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:58:56,137 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:58:59,654 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 49.31 % 2021-12-02 05:58:59,654 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 49.31 2021-12-02 05:58:59,670 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:58:59,671 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:58:59,671 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:59:06,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 49.72 % 2021-12-02 05:59:06,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:59:06,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:59:09,684 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 49.72 % 2021-12-02 05:59:09,684 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 49.72 2021-12-02 05:59:09,700 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:59:09,701 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:59:09,701 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:59:16,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 49.72 % 2021-12-02 05:59:16,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:59:16,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:59:19,714 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.13 % 2021-12-02 05:59:19,714 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 50.13 2021-12-02 05:59:19,727 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:59:19,730 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:59:19,730 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:59:26,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.13 % 2021-12-02 05:59:26,206 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:59:26,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:59:29,742 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.13 % 2021-12-02 05:59:29,743 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 50.13 2021-12-02 05:59:29,759 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:59:29,762 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:59:29,762 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:59:36,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.13 % 2021-12-02 05:59:36,227 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:59:36,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:59:39,775 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.13 % 2021-12-02 05:59:39,775 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 50.13 2021-12-02 05:59:39,792 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:59:39,796 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:59:39,796 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:59:46,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.13 % 2021-12-02 05:59:46,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:59:46,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:59:49,809 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.13 % 2021-12-02 05:59:49,809 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 50.13 2021-12-02 05:59:49,825 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:59:49,828 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:59:49,828 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:59:56,267 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.13 % 2021-12-02 05:59:56,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 05:59:56,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 05:59:59,844 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.13 % 2021-12-02 05:59:59,844 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 50.13 2021-12-02 05:59:59,864 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 05:59:59,865 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 05:59:59,865 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:00:06,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.55 % 2021-12-02 06:00:06,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:00:06,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:00:09,878 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.55 % 2021-12-02 06:00:09,878 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 50.55 2021-12-02 06:00:09,892 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:00:09,895 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:00:09,895 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:00:16,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.55 % 2021-12-02 06:00:16,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:00:16,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:00:19,907 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.55 % 2021-12-02 06:00:19,907 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 50.55 2021-12-02 06:00:19,924 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:00:19,927 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:00:19,927 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:00:26,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.55 % 2021-12-02 06:00:26,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:00:26,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:00:29,941 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.91 % 2021-12-02 06:00:29,943 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 50.91 2021-12-02 06:00:29,959 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:00:29,960 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:00:29,960 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:00:36,359 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.91 % 2021-12-02 06:00:36,372 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:00:36,374 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:00:39,974 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 51.33 % 2021-12-02 06:00:39,974 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 51.33 2021-12-02 06:00:39,990 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:00:39,992 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:00:39,992 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:00:46,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 51.33 % 2021-12-02 06:00:46,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:00:46,397 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:00:50,006 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 51.69 % 2021-12-02 06:00:50,006 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 51.69 2021-12-02 06:00:50,023 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:00:50,026 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:00:50,026 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:00:56,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 51.69 % 2021-12-02 06:00:56,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:00:56,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:01:00,046 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 51.69 % 2021-12-02 06:01:00,046 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 51.69 2021-12-02 06:01:00,065 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:01:00,069 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:01:00,069 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:01:06,428 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 51.69 % 2021-12-02 06:01:06,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:01:06,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:01:10,079 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 51.69 % 2021-12-02 06:01:10,081 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 51.69 2021-12-02 06:01:10,095 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:01:10,098 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:01:10,098 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:01:16,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 51.69 % 2021-12-02 06:01:16,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:01:16,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:01:20,112 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 51.69 % 2021-12-02 06:01:20,112 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 51.69 2021-12-02 06:01:20,131 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:01:20,134 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:01:20,134 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:01:26,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.16 % 2021-12-02 06:01:26,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:01:26,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:01:30,148 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.16 % 2021-12-02 06:01:30,148 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 52.16 2021-12-02 06:01:30,164 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:01:30,167 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:01:30,167 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:01:36,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.16 % 2021-12-02 06:01:36,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:01:36,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:01:40,180 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.16 % 2021-12-02 06:01:40,180 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 52.16 2021-12-02 06:01:40,196 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:01:40,198 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:01:40,198 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:01:46,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.16 % 2021-12-02 06:01:46,515 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:01:46,515 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:01:50,213 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.16 % 2021-12-02 06:01:50,214 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 52.16 2021-12-02 06:01:50,234 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:01:50,239 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:01:50,240 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:01:56,528 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.16 % 2021-12-02 06:01:56,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:01:56,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:02:00,250 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.16 % 2021-12-02 06:02:00,250 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 52.16 2021-12-02 06:02:00,266 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:02:00,267 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:02:00,267 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:02:06,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.16 % 2021-12-02 06:02:06,556 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:02:06,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:02:10,282 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.99 % 2021-12-02 06:02:10,282 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 52.99 2021-12-02 06:02:10,296 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:02:10,299 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:02:10,299 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:02:16,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.99 % 2021-12-02 06:02:16,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:02:16,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:02:20,311 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.99 % 2021-12-02 06:02:20,311 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 52.99 2021-12-02 06:02:20,325 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:02:20,329 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:02:20,329 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:02:26,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 52.99 % 2021-12-02 06:02:26,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:02:26,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:02:30,342 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.40 % 2021-12-02 06:02:30,344 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 53.4 2021-12-02 06:02:30,362 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:02:30,368 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:02:30,369 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:02:36,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.40 % 2021-12-02 06:02:36,627 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:02:36,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:02:40,384 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.40 % 2021-12-02 06:02:40,384 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 53.4 2021-12-02 06:02:40,400 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:02:40,404 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:02:40,404 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:02:46,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.40 % 2021-12-02 06:02:46,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:02:46,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:02:50,417 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.40 % 2021-12-02 06:02:50,417 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 53.4 2021-12-02 06:02:50,431 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:02:50,434 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:02:50,434 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:02:56,658 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.40 % 2021-12-02 06:02:56,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:02:56,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:03:00,446 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.81 % 2021-12-02 06:03:00,446 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 53.81 2021-12-02 06:03:00,460 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:03:00,461 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:03:00,461 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:03:06,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.81 % 2021-12-02 06:03:06,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:03:06,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:03:10,473 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.81 % 2021-12-02 06:03:10,473 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 53.81 2021-12-02 06:03:10,487 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:03:10,490 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:03:10,490 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:03:16,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.81 % 2021-12-02 06:03:16,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:03:16,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:03:20,506 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.81 % 2021-12-02 06:03:20,506 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 53.81 2021-12-02 06:03:20,526 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:03:20,528 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:03:20,528 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:03:26,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.81 % 2021-12-02 06:03:26,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:03:26,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:03:30,539 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 54.28 % 2021-12-02 06:03:30,540 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 54.28 2021-12-02 06:03:30,555 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:03:30,559 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:03:30,559 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:03:36,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 54.28 % 2021-12-02 06:03:36,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:03:36,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:03:40,572 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 54.28 % 2021-12-02 06:03:40,572 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 54.28 2021-12-02 06:03:40,588 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:03:40,592 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:03:40,592 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:03:46,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 54.28 % 2021-12-02 06:03:46,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:03:46,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:03:50,604 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 54.28 % 2021-12-02 06:03:50,604 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 54.28 2021-12-02 06:03:50,619 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:03:50,621 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:03:50,622 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:03:56,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.11 % 2021-12-02 06:03:56,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:03:56,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:04:00,644 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.11 % 2021-12-02 06:04:00,644 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 55.11 2021-12-02 06:04:00,661 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:04:00,664 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:04:00,664 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:04:06,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.11 % 2021-12-02 06:04:06,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:04:06,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:04:10,676 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.11 % 2021-12-02 06:04:10,676 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 55.11 2021-12-02 06:04:10,690 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:04:10,691 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:04:10,691 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:04:16,829 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.11 % 2021-12-02 06:04:16,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:04:16,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:04:20,706 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.11 % 2021-12-02 06:04:20,707 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 55.11 2021-12-02 06:04:20,720 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:04:20,721 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:04:20,721 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:04:26,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.11 % 2021-12-02 06:04:26,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:04:26,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:04:30,733 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.11 % 2021-12-02 06:04:30,733 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 55.11 2021-12-02 06:04:30,753 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:04:30,755 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:04:30,756 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:04:36,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.11 % 2021-12-02 06:04:36,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:04:36,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:04:40,767 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.11 % 2021-12-02 06:04:40,767 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 55.11 2021-12-02 06:04:40,786 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:04:40,789 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:04:40,789 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:04:46,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.52 % 2021-12-02 06:04:46,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:04:46,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:04:50,801 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.52 % 2021-12-02 06:04:50,801 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 55.52 2021-12-02 06:04:50,818 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:04:50,821 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:04:50,821 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:04:56,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.52 % 2021-12-02 06:04:56,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:04:56,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:05:00,834 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.52 % 2021-12-02 06:05:00,834 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 55.52 2021-12-02 06:05:00,848 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:05:00,851 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:05:00,851 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:05:06,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.52 % 2021-12-02 06:05:06,946 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:05:06,946 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:05:10,862 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.94 % 2021-12-02 06:05:10,862 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 55.94 2021-12-02 06:05:10,898 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:05:10,901 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:05:10,901 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:05:16,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.94 % 2021-12-02 06:05:16,966 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:05:16,966 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:05:20,914 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.94 % 2021-12-02 06:05:20,914 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 55.94 2021-12-02 06:05:20,934 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:05:20,937 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:05:20,937 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:05:26,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.40 % 2021-12-02 06:05:26,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:05:26,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:05:30,950 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.40 % 2021-12-02 06:05:30,950 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 56.4 2021-12-02 06:05:30,964 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:05:30,967 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:05:30,967 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:05:36,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.40 % 2021-12-02 06:05:37,005 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:05:37,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:05:40,979 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.87 % 2021-12-02 06:05:40,979 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 56.87 2021-12-02 06:05:40,994 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:05:40,997 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:05:40,997 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:05:47,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.87 % 2021-12-02 06:05:47,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:05:47,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:05:51,009 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.87 % 2021-12-02 06:05:51,009 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 56.87 2021-12-02 06:05:51,026 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:05:51,029 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:05:51,029 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:05:57,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.87 % 2021-12-02 06:05:57,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:05:57,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:06:01,039 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.87 % 2021-12-02 06:06:01,039 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 56.87 2021-12-02 06:06:01,053 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:06:01,056 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:06:01,056 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:06:07,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.87 % 2021-12-02 06:06:07,058 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:06:07,058 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:06:11,069 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.87 % 2021-12-02 06:06:11,069 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 56.87 2021-12-02 06:06:11,086 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:06:11,088 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:06:11,088 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:06:17,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.87 % 2021-12-02 06:06:17,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:06:17,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:06:21,101 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.87 % 2021-12-02 06:06:21,101 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 56.87 2021-12-02 06:06:21,118 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:06:21,121 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:06:21,121 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:06:27,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.87 % 2021-12-02 06:06:27,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:06:27,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:06:31,132 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.87 % 2021-12-02 06:06:31,134 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 56.87 2021-12-02 06:06:31,151 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:06:31,153 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:06:31,153 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:06:37,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 57.33 % 2021-12-02 06:06:37,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:06:37,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:06:41,165 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 57.33 % 2021-12-02 06:06:41,165 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 57.33 2021-12-02 06:06:41,184 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:06:41,187 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:06:41,187 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:06:47,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 57.33 % 2021-12-02 06:06:47,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:06:47,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:06:51,197 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 57.33 % 2021-12-02 06:06:51,198 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 57.33 2021-12-02 06:06:51,213 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:06:51,217 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:06:51,217 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:06:57,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 57.70 % 2021-12-02 06:06:57,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:06:57,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:07:01,229 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.11 % 2021-12-02 06:07:01,229 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 58.11 2021-12-02 06:07:01,246 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:07:01,249 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:07:01,249 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:07:07,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.11 % 2021-12-02 06:07:07,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:07:07,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:07:11,266 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.52 % 2021-12-02 06:07:11,266 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 58.52 2021-12-02 06:07:11,282 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:07:11,285 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:07:11,285 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:07:17,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.52 % 2021-12-02 06:07:17,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:07:17,209 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:07:21,311 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.52 % 2021-12-02 06:07:21,311 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 58.52 2021-12-02 06:07:21,326 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:07:21,328 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:07:21,328 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:07:27,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.52 % 2021-12-02 06:07:27,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:07:27,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:07:31,341 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.52 % 2021-12-02 06:07:31,341 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 58.52 2021-12-02 06:07:31,357 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:07:31,359 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:07:31,359 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:07:37,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.52 % 2021-12-02 06:07:37,255 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:07:37,255 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:07:41,371 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.52 % 2021-12-02 06:07:41,371 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 58.52 2021-12-02 06:07:41,387 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:07:41,390 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:07:41,390 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:07:47,267 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.52 % 2021-12-02 06:07:47,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:07:47,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:07:51,401 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.52 % 2021-12-02 06:07:51,401 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 58.52 2021-12-02 06:07:51,417 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:07:51,420 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:07:51,420 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:07:57,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.52 % 2021-12-02 06:07:57,308 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:07:57,308 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:08:01,431 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.52 % 2021-12-02 06:08:01,431 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 58.52 2021-12-02 06:08:01,447 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:08:01,448 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:08:01,450 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:08:07,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.99 % 2021-12-02 06:08:07,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:08:07,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:08:11,460 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.99 % 2021-12-02 06:08:11,461 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 58.99 2021-12-02 06:08:11,476 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:08:11,477 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:08:11,477 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:08:17,341 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.45 % 2021-12-02 06:08:17,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:08:17,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:08:21,490 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.45 % 2021-12-02 06:08:21,490 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 59.45 2021-12-02 06:08:21,506 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:08:21,509 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:08:21,509 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:08:27,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.45 % 2021-12-02 06:08:27,369 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:08:27,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:08:31,525 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.45 % 2021-12-02 06:08:31,525 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 59.45 2021-12-02 06:08:31,542 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:08:31,546 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:08:31,546 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:08:37,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.45 % 2021-12-02 06:08:37,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:08:37,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:08:41,559 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.82 % 2021-12-02 06:08:41,561 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 59.82 2021-12-02 06:08:41,576 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:08:41,578 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:08:41,578 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:08:47,400 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 60.28 % 2021-12-02 06:08:47,408 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:08:47,408 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:08:51,592 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 60.28 % 2021-12-02 06:08:51,592 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 60.28 2021-12-02 06:08:51,611 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:08:51,614 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:08:51,614 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:08:57,410 | test | INFO | MainThread | [rest_client:rebalance_reached:119] Rebalance reached >60.28% in 3226.55299997 seconds 2021-12-02 06:08:57,424 | test | DEBUG | MainThread | [rest_client:diag_eval:1453] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-02 06:08:57,424 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-02 06:08:57,427 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 06:08:57,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:08:57,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 06:08:57,765 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:08:57,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 06:08:57,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 06:08:57,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 06:08:57,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 06:08:57,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 06:08:57,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 06:08:57,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 06:08:57,871 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 06:08:57,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 06:08:57,934 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['285945 memcached\n'], []) 2021-12-02 06:08:57,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 06:08:57,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 06:08:58,023 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['285945 memcached\n'], []) 2021-12-02 06:08:58,023 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 06:08:58,028 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 06:08:58,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:08:58,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 06:08:58,385 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:08:58,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 06:08:58,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 06:08:58,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 06:08:58,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 06:08:58,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 06:08:58,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 06:08:58,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 06:08:58,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:08:58,490 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 06:08:58,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:08:58,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:08:58,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:08:58,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:08:58,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:08:58,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:08:58,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:08:58,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:08:58,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:08:58,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:08:58,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:08:58,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:08:58,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:08:58,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:08:58,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:08:58,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:08:58,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:08:58,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:08:58,901 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 06:08:58,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:08:58,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:08:58,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:08:58,936 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 06:08:58,937 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 06:08:58,940 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 06:08:59,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:08:59,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 06:08:59,394 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:08:59,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 06:08:59,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 06:08:59,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 06:08:59,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 06:08:59,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 06:08:59,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 06:08:59,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 06:08:59,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:08:59,549 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 06:08:59,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:08:59,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:08:59,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:08:59,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:08:59,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:08:59,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:08:59,713 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 06:08:59,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:08:59,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:08:59,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:08:59,759 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 06:08:59,759 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 06:08:59,763 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 06:08:59,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:09:00,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 06:09:00,207 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:09:00,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 06:09:00,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 06:09:00,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 06:09:00,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 06:09:00,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 06:09:00,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 06:09:00,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 06:09:00,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:09:00,390 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 06:09:00,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:09:00,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:09:00,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:09:00,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:09:00,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:09:00,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:09:00,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:09:00,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:09:00,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:09:00,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:09:00,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:09:00,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:09:00,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:09:00,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:09:00,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:09:00,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:09:00,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:09:00,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:09:01,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:09:01,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 06:09:01,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 06:09:01,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:09:01,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:09:01,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 06:09:01,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 06:09:01,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:09:01,190 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 06:09:01,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:09:01,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:09:01,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:09:01,260 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 06:09:01,260 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 06:09:01,265 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 06:09:01,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:09:01,625 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 60.28 % 2021-12-02 06:09:01,627 | test | INFO | pool-3-thread-22 | [task:check:384] Rebalance - status: running, progress: 60.28 2021-12-02 06:09:01,648 | infra | DEBUG | pool-3-thread-22 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:09:01,650 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:09:01,651 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:09:01,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 06:09:01,733 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:09:01,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 06:09:01,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 06:09:01,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 06:09:01,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 06:09:01,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 06:09:01,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 06:09:01,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 06:09:01,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:09:01,857 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 06:09:01,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:09:01,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:09:01,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:09:01,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:09:02,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:09:02,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:09:02,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:09:02,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:09:02,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:09:02,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:09:02,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:09:02,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:09:02,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:09:02,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:09:02,382 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 06:09:02,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:09:02,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:09:02,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:09:02,437 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 06:09:02,437 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 06:09:02,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-02 06:09:02,444 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-02 06:09:11,660 | test | ERROR | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1640] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'd261de36e5ac40dfd34ea40832417a10', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=faf0bfba853cc15d2d34cd721c028b51', u'status': u'notRunning'} - rebalance failed 2021-12-02 06:09:11,677 | test | INFO | pool-3-thread-22 | [rest_client:print_UI_logs:2789] Latest logs from UI on 172.23.110.65: 2021-12-02 06:09:11,677 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638454150050L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:09:10.050Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.31886.123>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,93,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.20098.118>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 93}},\n infinity]}}}}}.\nRebalance Operation Id = e66aab4525440aa2a2befd0068c07d29'} 2021-12-02 06:09:11,677 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638454150042L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:09:10.042Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1202 05:05:41.473266 286178 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-02 06:09:11,677 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638454150014L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:09:10.014Z', u'text': u'Worker <0.32660.123> (for action {move,\n {93,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.65\'],\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.31886.123>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 151}]},\n {mc_client_binary,\n select_bucket,\n 2,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 323}]},\n {ns_memcached_sockets_pool,\n take_socket,\n 2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 82}]},\n {async,\n \'-async_init/4-fun-1-\',\n 3,\n [{file,\n "src/async.erl"},\n {line,\n 191}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 93,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.20098.118>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 93}},\n infinity]}}}}'} 2021-12-02 06:09:11,678 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638454149997L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:09:09.997Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-02 06:09:11,678 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638450791230L, u'shortText': u'message', u'serverTime': u'2021-12-02T05:13:11.230Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-02 06:09:11,678 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638450791070L, u'shortText': u'message', u'serverTime': u'2021-12-02T05:13:11.070Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-02 06:09:11,678 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638450790854L, u'shortText': u'message', u'serverTime': u'2021-12-02T05:13:10.854Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.68',\n 'ns_1@172.23.110.66','ns_1@172.23.110.65'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = e66aab4525440aa2a2befd0068c07d29"} 2021-12-02 06:09:11,678 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638450768643L, u'shortText': u'message', u'serverTime': u'2021-12-02T05:12:48.643Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 408 seconds.'} 2021-12-02 06:09:11,680 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638450332912L, u'shortText': u'message', u'serverTime': u'2021-12-02T05:05:32.912Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.25374.117>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,119,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.24541.111>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 119}},\n infinity]}}}}}.\nRebalance Operation Id = 123bd92a27793fac1899a0e32eb2c05f'} 2021-12-02 06:09:11,680 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638450332886L, u'shortText': u'message', u'serverTime': u'2021-12-02T05:05:32.886Z', u'text': u'Worker <0.25346.117> (for action {move,\n {119,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.65\'],\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.25374.117>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 151}]},\n {mc_client_binary,\n select_bucket,\n 2,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 323}]},\n {ns_memcached_sockets_pool,\n take_socket,\n 2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 82}]},\n {async,\n \'-async_init/4-fun-1-\',\n 3,\n [{file,\n "src/async.erl"},\n {line,\n 191}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 119,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.24541.111>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 119}},\n infinity]}}}}'} 2021-12-02 06:09:11,680 | test | ERROR | pool-3-thread-22 | [task:call:296] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'd261de36e5ac40dfd34ea40832417a10', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=faf0bfba853cc15d2d34cd721c028b51', u'status': u'notRunning'} - rebalance failed 2021-12-02 06:10:02,457 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-02 06:10:02,457 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 06:10:02,470 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-02 06:10:02,486 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.68:8091 2021-12-02 06:10:02,486 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 06:10:02,497 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.68:8091 is running 2021-12-02 06:10:02,512 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-02 06:10:02,512 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 06:10:02,523 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:02,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:02,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:04,546 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:04,553 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:04,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:06,569 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:06,576 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:06,576 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:08,589 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:08,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:08,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:10,611 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:10,617 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:10,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:12,631 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:12,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:12,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:14,651 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:14,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:14,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:16,671 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:16,678 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:16,678 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:18,690 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:18,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:18,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:20,713 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:20,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:20,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:22,736 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:22,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:22,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:24,756 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:24,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:24,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:26,776 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:26,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:26,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:28,796 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:28,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:28,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:30,819 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:30,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:30,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:32,839 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:32,846 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:32,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:34,861 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:34,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:34,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:36,881 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:36,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:36,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:38,903 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:38,910 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:38,910 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:40,937 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:40,946 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:40,946 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:42,957 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:42,964 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:42,964 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:44,976 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:44,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:44,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:46,996 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:47,002 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:47,002 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:49,013 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:49,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:49,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:51,032 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:51,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:51,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:53,049 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:53,058 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:53,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:55,072 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:55,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:55,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:57,095 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:57,102 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:57,102 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:10:59,117 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:10:59,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:10:59,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:01,144 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:01,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:01,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:03,167 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:03,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:03,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:05,187 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:05,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:05,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:07,209 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:07,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:07,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:09,229 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:09,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:09,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:11,260 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:11,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:11,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:13,282 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:13,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:13,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:15,302 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:15,308 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:15,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:17,319 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:17,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:17,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:19,342 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:19,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:19,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:21,361 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:21,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:21,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:23,385 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:23,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:23,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:25,410 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:25,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:25,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:27,430 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:27,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:27,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:29,448 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:29,456 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:29,456 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:31,467 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:31,474 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:31,474 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:33,487 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:33,494 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:33,494 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:35,509 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:35,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:35,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:37,532 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:37,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:37,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:39,552 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:39,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:39,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:41,572 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:41,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:41,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:43,591 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:43,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:43,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:45,611 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:45,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:45,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:47,632 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:47,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:47,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:49,654 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:49,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:49,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:51,676 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:51,681 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:51,683 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:53,694 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:53,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:53,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:55,714 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:55,721 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:55,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:57,734 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:57,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:57,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:11:59,756 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:11:59,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:11:59,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:01,776 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:01,785 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:01,785 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:03,799 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:03,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:03,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:05,821 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:05,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:05,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:07,845 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:07,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:07,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:09,865 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:09,872 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:09,872 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:11,885 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:11,892 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:11,892 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:13,905 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:13,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:13,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:15,927 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:15,936 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:15,936 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:17,950 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:17,959 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:17,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:19,973 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:19,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:19,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:21,993 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:22,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:22,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:24,013 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:24,020 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:24,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:26,033 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:26,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:26,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:28,052 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:28,058 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:28,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:30,072 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:30,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:30,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:32,092 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:32,099 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:32,099 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:34,112 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:34,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:34,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:36,135 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:36,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:36,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:38,158 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:38,165 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:38,165 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:40,180 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:40,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:40,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:42,203 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:42,209 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:42,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:44,223 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:44,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:44,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:46,244 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:46,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:46,252 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:48,269 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:48,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:48,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:50,289 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:50,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:50,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:52,309 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:52,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:52,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:54,331 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:54,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:54,336 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:56,352 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:56,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:56,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:12:58,375 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:12:58,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:12:58,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:00,397 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:00,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:00,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:02,417 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:02,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:02,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:04,438 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:04,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:04,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:06,461 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:06,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:06,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:08,486 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:08,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:08,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:10,507 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:10,515 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:10,515 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:12,526 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:12,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:12,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:14,548 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:14,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:14,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:16,568 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:16,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:16,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:18,588 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:18,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:18,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:20,609 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:20,617 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:20,617 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:22,628 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:22,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:22,637 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:24,648 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:24,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:24,655 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:26,671 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:26,678 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:26,678 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:28,693 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:28,698 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:28,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:30,713 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:30,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:30,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:32,733 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:32,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:32,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:34,753 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:34,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:34,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:36,773 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:36,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:36,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:38,790 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:38,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:38,798 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:40,809 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:40,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:40,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:42,831 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:42,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:42,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:44,851 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:44,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:44,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:46,872 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:46,880 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:46,880 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:48,892 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:48,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:48,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:50,915 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:50,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:50,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:52,938 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:52,946 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:52,946 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:54,960 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:54,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:54,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:56,983 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:56,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:56,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:13:59,007 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:13:59,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:13:59,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:01,030 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:01,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:01,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:03,049 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:03,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:03,056 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:05,071 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:05,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:05,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:07,092 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:07,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:07,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:09,111 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:09,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:09,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:11,131 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:11,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:11,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:13,153 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:13,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:13,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:15,173 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:15,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:15,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:17,193 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:17,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:17,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:19,213 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:19,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:19,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:21,237 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:21,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:21,252 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:23,267 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:23,275 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:23,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:25,288 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:25,293 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:25,295 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:27,309 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:27,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:27,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:29,328 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:29,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:29,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:31,348 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:31,354 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:31,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:33,367 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:33,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:33,374 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:35,388 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:35,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:35,395 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:37,411 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:37,420 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:37,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:39,433 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:39,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:39,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:41,454 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:41,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:41,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:43,473 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:43,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:43,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:45,497 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:45,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:45,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:47,545 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:47,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:47,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:49,565 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:49,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:49,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:51,584 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:51,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:51,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:53,605 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:53,612 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:53,612 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:55,627 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:55,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:55,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:57,650 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:57,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:57,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:14:59,673 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:14:59,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:14:59,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:15:01,694 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:15:01,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:15:01,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:15:03,719 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:15:03,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:15:03,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:15:05,739 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:15:05,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:15:05,747 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:15:07,759 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:15:07,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:15:07,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:15:12,190 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:15:12,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:15:12,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:15:14,217 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:15:14,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:15:14,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:15:16,237 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:15:16,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:15:16,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:15:18,260 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:15:18,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:15:18,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:15:20,285 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:15:20,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:15:20,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:15:22,305 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:15:22,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:15:22,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:15:24,323 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:15:24,331 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:15:24,331 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:15:26,342 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-02 06:15:26,354 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-02 06:15:26,354 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 06:15:26,365 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-02 06:15:26,367 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 06:15:26,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:15:26,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 06:15:26,706 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:15:26,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 06:15:26,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 06:15:26,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 06:15:26,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 06:15:26,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 06:15:26,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 06:15:26,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 06:15:26,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:15:26,796 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 06:15:26,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:15:26,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:26,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:26,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:15:26,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:15:26,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:26,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:27,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:15:27,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:15:27,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:27,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:27,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:15:27,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:15:27,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:27,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:27,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:15:27,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:15:27,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:15:27,223 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 06:15:27,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:15:27,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:27,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:15:27,263 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 06:15:27,263 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 06:15:27,267 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 06:15:27,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:15:27,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 06:15:27,681 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:15:27,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 06:15:27,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 06:15:27,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 06:15:27,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 06:15:27,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 06:15:27,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 06:15:27,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 06:15:27,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:15:27,836 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 06:15:27,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:15:27,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:27,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:27,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:15:27,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:15:28,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:15:28,039 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 06:15:28,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:15:28,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:28,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:15:28,092 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 06:15:28,092 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 06:15:28,095 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 06:15:28,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:15:28,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 06:15:28,505 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:15:28,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 06:15:28,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 06:15:28,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 06:15:28,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 06:15:28,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 06:15:28,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 06:15:28,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 06:15:28,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:15:28,647 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 06:15:28,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:15:28,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:28,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:28,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:15:28,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:15:28,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:28,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:28,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:15:29,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:15:29,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:29,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:29,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:15:29,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:15:29,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:29,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:29,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:15:29,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:15:29,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:29,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:29,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 06:15:29,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 06:15:29,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:29,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:29,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 06:15:29,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 06:15:29,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:15:29,447 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 06:15:29,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:15:29,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:29,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:15:29,486 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 06:15:29,486 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 06:15:29,490 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 06:15:29,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:15:29,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 06:15:29,934 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:15:29,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 06:15:29,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 06:15:29,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 06:15:30,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 06:15:30,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 06:15:30,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 06:15:30,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 06:15:30,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:15:30,115 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 06:15:30,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:15:30,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:30,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:30,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:15:30,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:15:30,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:30,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:30,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:15:30,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:15:30,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:30,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:15:30,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:15:30,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:15:30,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:15:30,463 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 06:15:30,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:15:30,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:15:30,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:15:30,496 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 06:15:30,496 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 06:15:30,496 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1638450790.79 2021-12-02 06:15:30,496 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 60 2021-12-02 06:15:30,523 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1638454530.5 2021-12-02 06:15:30,523 | infra | INFO | pool-3-thread-23 | [task:start_task:89] Thread 'Rebalance_task_IN=[]_OUT=[]_1638454530.5' started 2021-12-02 06:15:30,530 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-02 06:15:30,530 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-02 06:15:30,551 | test | DEBUG | pool-3-thread-23 | [rest_client:rebalance:1431] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-02 06:15:30,604 | test | DEBUG | pool-3-thread-23 | [rest_client:rebalance:1436] Rebalance operation started 2021-12-02 06:15:30,607 | test | INFO | pool-3-thread-23 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1817-enterprise | 5.08447729673 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1817-enterprise | 0.437761440861 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1817-enterprise | 0.923081200562 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1817-enterprise | 1.25410155164 | Cluster node | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-02 06:15:30,615 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:15:30,615 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:15:30,630 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:15:30,631 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:15:30,631 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:15:40,642 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:15:40,644 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:15:40,657 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:15:40,658 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:15:40,658 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:15:50,670 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:15:50,670 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:15:50,688 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:15:50,690 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:15:50,690 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:16:00,700 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:16:00,700 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:16:00,714 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:16:00,717 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:16:00,717 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:16:10,727 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:16:10,729 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:16:10,742 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:16:10,743 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:16:10,743 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:16:20,753 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:16:20,755 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:16:20,767 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:16:20,770 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:16:20,770 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:16:30,782 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:16:30,782 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:16:30,802 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:16:30,805 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:16:30,805 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:16:40,816 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:16:40,816 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:16:40,835 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:16:40,836 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:16:40,836 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:16:50,848 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.70 % 2021-12-02 06:16:50,848 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 0.7 2021-12-02 06:16:50,867 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:16:50,868 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:16:50,869 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:17:00,878 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.40 % 2021-12-02 06:17:00,880 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 2.4 2021-12-02 06:17:00,892 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:17:00,894 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:17:00,894 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:17:10,907 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.40 % 2021-12-02 06:17:10,907 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 2.4 2021-12-02 06:17:10,926 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:17:10,928 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:17:10,928 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:17:20,940 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.35 % 2021-12-02 06:17:20,941 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 5.35 2021-12-02 06:17:20,960 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:17:20,963 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:17:20,963 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:17:30,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.35 % 2021-12-02 06:17:30,542 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 5.35 2021-12-02 06:17:30,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.35 % 2021-12-02 06:17:30,556 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:17:30,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:17:30,973 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.35 % 2021-12-02 06:17:30,973 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 5.35 2021-12-02 06:17:30,994 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:17:30,996 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:17:30,996 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:17:40,568 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.35 % 2021-12-02 06:17:40,573 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:17:40,573 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:17:41,009 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.35 % 2021-12-02 06:17:41,009 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 5.35 2021-12-02 06:17:41,030 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:17:41,033 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:17:41,033 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:17:50,584 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.35 % 2021-12-02 06:17:50,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:17:50,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:17:51,046 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.35 % 2021-12-02 06:17:51,046 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 5.35 2021-12-02 06:17:51,063 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:17:51,066 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:17:51,066 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:18:00,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.05 % 2021-12-02 06:18:00,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:18:00,604 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:18:01,078 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.05 % 2021-12-02 06:18:01,079 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 6.05 2021-12-02 06:18:01,104 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:18:01,107 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:18:01,107 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:18:10,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.05 % 2021-12-02 06:18:10,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:18:10,619 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:18:11,117 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.05 % 2021-12-02 06:18:11,117 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 6.05 2021-12-02 06:18:11,132 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:18:11,135 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:18:11,135 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:18:20,630 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.05 % 2021-12-02 06:18:20,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:18:20,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:18:21,145 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.20 % 2021-12-02 06:18:21,147 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 7.2 2021-12-02 06:18:21,161 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:18:21,164 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:18:21,164 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:18:30,645 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.20 % 2021-12-02 06:18:30,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:18:30,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:18:31,177 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.20 % 2021-12-02 06:18:31,178 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 7.2 2021-12-02 06:18:31,194 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:18:31,197 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:18:31,197 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:18:40,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.20 % 2021-12-02 06:18:40,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:18:40,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:18:41,207 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.20 % 2021-12-02 06:18:41,207 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 7.2 2021-12-02 06:18:41,223 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:18:41,226 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:18:41,226 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:18:50,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 06:18:50,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:18:50,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:18:51,239 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 06:18:51,239 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 10.15 2021-12-02 06:18:51,256 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:18:51,259 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:18:51,259 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:19:00,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 06:19:00,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:19:00,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:19:01,269 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 06:19:01,269 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 10.15 2021-12-02 06:19:01,286 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:19:01,289 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:19:01,289 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:19:10,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 06:19:10,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:19:10,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:19:11,299 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 06:19:11,299 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 10.15 2021-12-02 06:19:11,315 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:19:11,316 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:19:11,318 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:19:20,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 06:19:20,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:19:20,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:19:21,328 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 06:19:21,329 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 10.15 2021-12-02 06:19:21,345 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:19:21,348 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:19:21,348 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:19:30,742 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 06:19:30,747 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:19:30,747 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:19:31,362 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 06:19:31,362 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 10.15 2021-12-02 06:19:31,380 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:19:31,381 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:19:31,382 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:19:40,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 06:19:40,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:19:40,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:19:41,392 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 06:19:41,392 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 10.15 2021-12-02 06:19:41,407 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:19:41,410 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:19:41,411 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:19:50,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.85 % 2021-12-02 06:19:50,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:19:50,780 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:19:51,421 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.54 % 2021-12-02 06:19:51,421 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 12.54 2021-12-02 06:19:51,437 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:19:51,441 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:19:51,441 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:20:00,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.54 % 2021-12-02 06:20:00,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:20:00,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:20:01,453 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.54 % 2021-12-02 06:20:01,454 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 12.54 2021-12-02 06:20:01,470 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:20:01,473 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:20:01,473 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:20:10,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.24 % 2021-12-02 06:20:10,813 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:20:10,813 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:20:11,483 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.24 % 2021-12-02 06:20:11,483 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 13.24 2021-12-02 06:20:11,497 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:20:11,500 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:20:11,500 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:20:20,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.94 % 2021-12-02 06:20:20,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:20:20,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:20:21,512 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.94 % 2021-12-02 06:20:21,513 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 14.94 2021-12-02 06:20:21,526 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:20:21,529 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:20:21,529 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:20:30,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.94 % 2021-12-02 06:20:30,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:20:30,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:20:31,539 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.94 % 2021-12-02 06:20:31,539 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 14.94 2021-12-02 06:20:31,556 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:20:31,559 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:20:31,559 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:20:40,859 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.94 % 2021-12-02 06:20:40,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:20:40,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:20:41,569 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.94 % 2021-12-02 06:20:41,569 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 14.94 2021-12-02 06:20:41,585 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:20:41,588 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:20:41,588 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:20:50,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.94 % 2021-12-02 06:20:50,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:20:50,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:20:51,599 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.94 % 2021-12-02 06:20:51,599 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 14.94 2021-12-02 06:20:51,614 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:20:51,615 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:20:51,615 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:21:00,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.94 % 2021-12-02 06:21:00,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:21:00,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:21:01,627 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.94 % 2021-12-02 06:21:01,627 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 14.94 2021-12-02 06:21:01,641 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:21:01,641 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:21:01,642 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:21:10,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.94 % 2021-12-02 06:21:10,918 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:21:10,918 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:21:11,654 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.94 % 2021-12-02 06:21:11,654 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 14.94 2021-12-02 06:21:11,668 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:21:11,671 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:21:11,671 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:21:20,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.64 % 2021-12-02 06:21:20,936 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:21:20,936 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:21:21,681 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.64 % 2021-12-02 06:21:21,681 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 15.64 2021-12-02 06:21:21,703 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:21:21,706 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:21:21,706 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:21:30,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.64 % 2021-12-02 06:21:30,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:21:30,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:21:31,717 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.64 % 2021-12-02 06:21:31,717 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 15.64 2021-12-02 06:21:31,732 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:21:31,733 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:21:31,733 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:21:40,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.59 % 2021-12-02 06:21:40,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:21:40,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:21:41,744 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.59 % 2021-12-02 06:21:41,744 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 18.59 2021-12-02 06:21:41,759 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:21:41,762 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:21:41,762 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:21:50,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.59 % 2021-12-02 06:21:50,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:21:50,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:21:51,772 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.59 % 2021-12-02 06:21:51,773 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 18.59 2021-12-02 06:21:51,785 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:21:51,786 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:21:51,788 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:22:00,999 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.74 % 2021-12-02 06:22:01,005 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:22:01,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:22:01,798 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.74 % 2021-12-02 06:22:01,799 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 19.74 2021-12-02 06:22:01,812 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:22:01,815 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:22:01,815 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:22:11,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.74 % 2021-12-02 06:22:11,022 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:22:11,022 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:22:11,828 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.74 % 2021-12-02 06:22:11,828 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 19.74 2021-12-02 06:22:11,842 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:22:11,845 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:22:11,845 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:22:21,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.74 % 2021-12-02 06:22:21,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:22:21,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:22:21,855 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.74 % 2021-12-02 06:22:21,855 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 19.74 2021-12-02 06:22:21,875 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:22:21,878 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:22:21,878 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:22:31,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.74 % 2021-12-02 06:22:31,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:22:31,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:22:31,888 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.74 % 2021-12-02 06:22:31,890 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 19.74 2021-12-02 06:22:31,903 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:22:31,904 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:22:31,904 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:22:41,073 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.74 % 2021-12-02 06:22:41,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:22:41,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:22:41,917 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.74 % 2021-12-02 06:22:41,917 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 19.74 2021-12-02 06:22:41,938 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:22:41,940 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:22:41,940 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:22:51,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.74 % 2021-12-02 06:22:51,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:22:51,102 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:22:51,953 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.74 % 2021-12-02 06:22:51,953 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 19.74 2021-12-02 06:22:51,973 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:22:51,976 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:22:51,976 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:23:01,112 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.44 % 2021-12-02 06:23:01,122 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:23:01,124 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:23:01,986 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.44 % 2021-12-02 06:23:01,986 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 20.44 2021-12-02 06:23:02,007 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:23:02,009 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:23:02,009 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:23:11,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.39 % 2021-12-02 06:23:11,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:23:11,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:23:12,022 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.39 % 2021-12-02 06:23:12,022 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 23.39 2021-12-02 06:23:12,036 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:23:12,039 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:23:12,039 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:23:21,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.39 % 2021-12-02 06:23:21,188 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:23:21,188 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:23:22,049 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.39 % 2021-12-02 06:23:22,051 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 23.39 2021-12-02 06:23:22,068 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:23:22,072 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:23:22,072 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:23:31,204 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:23:31,221 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:23:31,221 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:23:32,084 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:23:32,085 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 25.09 2021-12-02 06:23:32,101 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:23:32,105 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:23:32,105 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:23:41,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:23:41,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:23:41,247 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:23:42,117 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:23:42,117 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 25.09 2021-12-02 06:23:42,134 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:23:42,137 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:23:42,138 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:23:51,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:23:51,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:23:51,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:23:52,148 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:23:52,148 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 25.09 2021-12-02 06:23:52,164 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:23:52,167 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:23:52,167 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:24:01,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:24:01,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:24:01,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:24:02,177 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:24:02,177 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 25.09 2021-12-02 06:24:02,197 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:24:02,200 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:24:02,200 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:24:11,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:24:11,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:24:11,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:24:12,213 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:24:12,213 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 25.09 2021-12-02 06:24:12,229 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:24:12,246 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:24:12,246 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:24:21,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:24:21,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:24:21,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:24:22,256 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:24:22,256 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 25.09 2021-12-02 06:24:22,270 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:24:22,273 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:24:22,275 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:24:31,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:24:31,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:24:31,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:24:32,298 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.09 % 2021-12-02 06:24:32,299 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 25.09 2021-12-02 06:24:32,312 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:24:32,315 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:24:32,315 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:24:41,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.04 % 2021-12-02 06:24:41,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:24:41,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:24:42,332 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.04 % 2021-12-02 06:24:42,332 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 27.04 2021-12-02 06:24:42,348 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:24:42,351 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:24:42,351 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:24:51,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.74 % 2021-12-02 06:24:51,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:24:51,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:24:52,369 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 27.74 % 2021-12-02 06:24:52,369 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 27.74 2021-12-02 06:24:52,385 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:24:52,388 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:24:52,388 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:25:01,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:25:01,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:25:01,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:25:02,401 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:25:02,401 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 29.44 2021-12-02 06:25:02,417 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:25:02,421 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:25:02,421 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:25:11,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:25:11,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:25:11,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:25:12,434 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:25:12,434 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 29.44 2021-12-02 06:25:12,450 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:25:12,453 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:25:12,453 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:25:21,446 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:25:21,456 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:25:21,456 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:25:22,463 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:25:22,464 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 29.44 2021-12-02 06:25:22,483 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:25:22,487 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:25:22,487 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:25:31,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:25:31,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:25:31,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:25:32,497 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:25:32,499 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 29.44 2021-12-02 06:25:32,513 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:25:32,516 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:25:32,516 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:25:41,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:25:41,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:25:41,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:25:42,529 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:25:42,529 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 29.44 2021-12-02 06:25:42,545 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:25:42,548 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:25:42,548 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:25:51,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:25:51,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:25:51,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:25:52,561 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:25:52,561 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 29.44 2021-12-02 06:25:52,576 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:25:52,579 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:25:52,579 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:26:01,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:26:01,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:26:01,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:26:02,592 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 29.44 % 2021-12-02 06:26:02,592 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 29.44 2021-12-02 06:26:02,608 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:26:02,611 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:26:02,611 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:26:11,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.29 % 2021-12-02 06:26:11,563 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:26:11,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:26:12,622 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 31.29 % 2021-12-02 06:26:12,622 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 31.29 2021-12-02 06:26:12,638 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:26:12,640 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:26:12,640 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:26:21,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.54 % 2021-12-02 06:26:21,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:26:21,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:26:22,650 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.54 % 2021-12-02 06:26:22,650 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 32.54 2021-12-02 06:26:22,664 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:26:22,664 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:26:22,664 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:26:31,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.24 % 2021-12-02 06:26:31,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:26:31,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:26:32,677 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.24 % 2021-12-02 06:26:32,678 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 34.24 2021-12-02 06:26:32,691 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:26:32,694 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:26:32,694 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:26:41,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.24 % 2021-12-02 06:26:41,630 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:26:41,630 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:26:42,704 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.24 % 2021-12-02 06:26:42,706 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 34.24 2021-12-02 06:26:42,724 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:26:42,726 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:26:42,726 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:26:51,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.24 % 2021-12-02 06:26:51,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:26:51,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:26:52,737 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.24 % 2021-12-02 06:26:52,737 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 34.24 2021-12-02 06:26:52,750 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:26:52,753 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:26:52,753 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:27:01,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.24 % 2021-12-02 06:27:01,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:27:01,668 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:27:02,766 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.24 % 2021-12-02 06:27:02,766 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 34.24 2021-12-02 06:27:02,779 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:27:02,782 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:27:02,782 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:27:11,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.24 % 2021-12-02 06:27:11,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:27:11,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:27:12,792 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 34.24 % 2021-12-02 06:27:12,792 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 34.24 2021-12-02 06:27:12,806 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:27:12,809 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:27:12,809 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:27:21,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.94 % 2021-12-02 06:27:21,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:27:21,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:27:22,822 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.94 % 2021-12-02 06:27:22,822 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 35.94 2021-12-02 06:27:22,841 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:27:22,842 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:27:22,842 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:27:31,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.94 % 2021-12-02 06:27:31,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:27:31,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:27:32,854 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.94 % 2021-12-02 06:27:32,854 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 35.94 2021-12-02 06:27:32,875 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:27:32,877 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:27:32,877 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:27:41,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.94 % 2021-12-02 06:27:41,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:27:41,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:27:42,888 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.94 % 2021-12-02 06:27:42,888 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 35.94 2021-12-02 06:27:42,903 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:27:42,904 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:27:42,904 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:27:51,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.89 % 2021-12-02 06:27:51,749 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:27:51,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:27:52,915 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.89 % 2021-12-02 06:27:52,915 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 37.89 2021-12-02 06:27:52,928 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:27:52,930 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:27:52,930 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:28:01,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.89 % 2021-12-02 06:28:01,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:28:01,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:28:02,943 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.89 % 2021-12-02 06:28:02,943 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 37.89 2021-12-02 06:28:02,963 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:28:02,964 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:28:02,966 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:28:11,778 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.58 % 2021-12-02 06:28:11,783 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:28:11,783 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:28:12,976 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.58 % 2021-12-02 06:28:12,976 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 38.58 2021-12-02 06:28:12,996 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:28:12,997 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:28:12,997 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:28:21,793 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.58 % 2021-12-02 06:28:21,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:28:21,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:28:23,009 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.58 % 2021-12-02 06:28:23,009 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 38.58 2021-12-02 06:28:23,023 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:28:23,026 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:28:23,026 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:28:31,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.58 % 2021-12-02 06:28:31,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:28:31,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:28:33,036 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.58 % 2021-12-02 06:28:33,036 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 38.58 2021-12-02 06:28:33,052 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:28:33,053 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:28:33,053 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:28:41,823 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.58 % 2021-12-02 06:28:41,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:28:41,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:28:43,065 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.58 % 2021-12-02 06:28:43,065 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 38.58 2021-12-02 06:28:43,082 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:28:43,085 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:28:43,085 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:28:51,841 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.58 % 2021-12-02 06:28:51,846 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:28:51,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:28:53,098 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 38.58 % 2021-12-02 06:28:53,098 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 38.58 2021-12-02 06:28:53,118 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:28:53,119 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:28:53,121 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:29:01,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.28 % 2021-12-02 06:29:01,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:29:01,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:29:03,131 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.28 % 2021-12-02 06:29:03,131 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 39.28 2021-12-02 06:29:03,150 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:29:03,153 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:29:03,153 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:29:11,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.98 % 2021-12-02 06:29:11,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:29:11,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:29:13,165 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 40.98 % 2021-12-02 06:29:13,165 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 40.98 2021-12-02 06:29:13,184 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:29:13,186 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:29:13,187 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:29:21,892 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.68 % 2021-12-02 06:29:21,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:29:21,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:29:23,197 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.68 % 2021-12-02 06:29:23,197 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 42.68 2021-12-02 06:29:23,211 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:29:23,213 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:29:23,213 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:29:31,910 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.68 % 2021-12-02 06:29:31,915 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:29:31,915 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:29:33,224 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.68 % 2021-12-02 06:29:33,224 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 42.68 2021-12-02 06:29:33,240 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:29:33,243 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:29:33,243 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:29:41,926 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.68 % 2021-12-02 06:29:41,933 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:29:41,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:29:43,253 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.68 % 2021-12-02 06:29:43,255 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 42.68 2021-12-02 06:29:43,267 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:29:43,270 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:29:43,270 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:29:51,943 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.68 % 2021-12-02 06:29:51,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:29:51,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:29:53,280 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.68 % 2021-12-02 06:29:53,280 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 42.68 2021-12-02 06:29:53,296 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:29:53,299 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:29:53,299 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:30:01,959 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.93 % 2021-12-02 06:30:01,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:30:01,964 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:30:03,309 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.93 % 2021-12-02 06:30:03,309 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 43.93 2021-12-02 06:30:03,325 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:30:03,328 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:30:03,328 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:30:11,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.93 % 2021-12-02 06:30:11,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:30:11,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:30:13,338 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.93 % 2021-12-02 06:30:13,338 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 43.93 2021-12-02 06:30:13,358 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:30:13,361 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:30:13,361 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:30:21,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 43.93 % 2021-12-02 06:30:22,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:30:22,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:30:23,371 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 44.63 % 2021-12-02 06:30:23,371 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 44.63 2021-12-02 06:30:23,385 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:30:23,388 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:30:23,388 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:30:32,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.78 % 2021-12-02 06:30:32,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:30:32,017 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:30:33,401 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.78 % 2021-12-02 06:30:33,401 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 45.78 2021-12-02 06:30:33,417 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:30:33,420 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:30:33,420 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:30:42,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.78 % 2021-12-02 06:30:42,035 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:30:42,035 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:30:43,430 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.78 % 2021-12-02 06:30:43,430 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 45.78 2021-12-02 06:30:43,444 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:30:43,447 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:30:43,447 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:30:52,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.78 % 2021-12-02 06:30:52,051 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:30:52,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:30:53,460 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.78 % 2021-12-02 06:30:53,460 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 45.78 2021-12-02 06:30:53,474 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:30:53,477 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:30:53,477 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:31:02,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.78 % 2021-12-02 06:31:02,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:31:02,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:31:03,489 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.78 % 2021-12-02 06:31:03,489 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 45.78 2021-12-02 06:31:03,505 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:31:03,507 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:31:03,507 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:31:12,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 45.78 % 2021-12-02 06:31:12,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:31:12,084 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:31:13,519 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 47.48 % 2021-12-02 06:31:13,519 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 47.48 2021-12-02 06:31:13,532 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:31:13,536 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:31:13,536 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:31:22,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 47.48 % 2021-12-02 06:31:22,102 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:31:22,102 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:31:23,546 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 47.48 % 2021-12-02 06:31:23,548 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 47.48 2021-12-02 06:31:23,563 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:31:23,566 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:31:23,566 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:31:32,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 47.48 % 2021-12-02 06:31:32,122 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:31:32,122 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:31:33,596 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 47.48 % 2021-12-02 06:31:33,596 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 47.48 2021-12-02 06:31:33,617 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:31:33,619 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:31:33,621 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:31:42,132 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.63 % 2021-12-02 06:31:42,142 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:31:42,142 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:31:43,631 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 48.63 % 2021-12-02 06:31:43,631 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 48.63 2021-12-02 06:31:43,647 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:31:43,651 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:31:43,651 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:31:52,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 49.88 % 2021-12-02 06:31:52,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:31:52,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:31:53,661 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 49.88 % 2021-12-02 06:31:53,661 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 49.88 2021-12-02 06:31:53,677 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:31:53,678 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:31:53,678 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:32:02,176 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 49.88 % 2021-12-02 06:32:02,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:32:02,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:32:03,690 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 49.88 % 2021-12-02 06:32:03,690 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 49.88 2021-12-02 06:32:03,704 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:32:03,707 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:32:03,707 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:32:12,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.58 % 2021-12-02 06:32:12,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:32:12,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:32:13,717 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.58 % 2021-12-02 06:32:13,717 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 50.58 2021-12-02 06:32:13,733 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:32:13,734 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:32:13,736 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:32:22,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.58 % 2021-12-02 06:32:22,232 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:32:22,232 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:32:23,747 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.58 % 2021-12-02 06:32:23,747 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 50.58 2021-12-02 06:32:23,765 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:32:23,766 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:32:23,766 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:32:32,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.58 % 2021-12-02 06:32:32,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:32:32,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:32:33,776 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.58 % 2021-12-02 06:32:33,776 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 50.58 2021-12-02 06:32:33,790 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:32:33,792 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:32:33,792 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:32:42,260 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.58 % 2021-12-02 06:32:42,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:32:42,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:32:43,803 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.58 % 2021-12-02 06:32:43,803 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 50.58 2021-12-02 06:32:43,819 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:32:43,821 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:32:43,821 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:32:52,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.58 % 2021-12-02 06:32:52,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:32:52,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:32:53,832 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.58 % 2021-12-02 06:32:53,832 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 50.58 2021-12-02 06:32:53,845 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:32:53,848 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:32:53,848 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:33:02,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.58 % 2021-12-02 06:33:02,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:33:02,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:33:03,858 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 50.58 % 2021-12-02 06:33:03,858 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 50.58 2021-12-02 06:33:03,871 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:33:03,874 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:33:03,874 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:33:12,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.43 % 2021-12-02 06:33:12,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:33:12,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:33:13,884 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.43 % 2021-12-02 06:33:13,884 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 53.43 2021-12-02 06:33:13,900 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:33:13,901 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:33:13,901 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:33:22,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.43 % 2021-12-02 06:33:22,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:33:22,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:33:23,913 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 53.43 % 2021-12-02 06:33:23,913 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 53.43 2021-12-02 06:33:23,931 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:33:23,934 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:33:23,934 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:33:32,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 54.13 % 2021-12-02 06:33:32,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:33:32,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:33:33,944 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 54.13 % 2021-12-02 06:33:33,944 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 54.13 2021-12-02 06:33:33,957 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:33:33,960 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:33:33,960 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:33:42,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.38 % 2021-12-02 06:33:42,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:33:42,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:33:43,971 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.38 % 2021-12-02 06:33:43,971 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 55.38 2021-12-02 06:33:43,986 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:33:43,989 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:33:43,989 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:33:52,403 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.38 % 2021-12-02 06:33:52,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:33:52,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:33:54,000 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.38 % 2021-12-02 06:33:54,000 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 55.38 2021-12-02 06:33:54,015 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:33:54,016 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:33:54,017 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:34:02,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.38 % 2021-12-02 06:34:02,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:34:02,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:34:04,028 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.38 % 2021-12-02 06:34:04,029 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 55.38 2021-12-02 06:34:04,048 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:34:04,049 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:34:04,049 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:34:12,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.38 % 2021-12-02 06:34:12,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:34:12,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:34:14,062 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.38 % 2021-12-02 06:34:14,062 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 55.38 2021-12-02 06:34:14,082 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:34:14,084 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:34:14,084 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:34:22,477 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.38 % 2021-12-02 06:34:22,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:34:22,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:34:24,095 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.38 % 2021-12-02 06:34:24,095 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 55.38 2021-12-02 06:34:24,115 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:34:24,118 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:34:24,119 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:34:32,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 55.38 % 2021-12-02 06:34:32,515 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:34:32,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:34:34,130 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 56.63 % 2021-12-02 06:34:34,131 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 56.63 2021-12-02 06:34:34,144 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:34:34,148 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:34:34,148 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:34:42,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 57.78 % 2021-12-02 06:34:42,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:34:42,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:34:44,160 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 57.78 % 2021-12-02 06:34:44,160 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 57.78 2021-12-02 06:34:44,177 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:34:44,180 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:34:44,180 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:34:52,545 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 57.78 % 2021-12-02 06:34:52,556 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:34:52,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:34:54,191 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 57.78 % 2021-12-02 06:34:54,191 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 57.78 2021-12-02 06:34:54,209 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:34:54,211 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:34:54,211 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:35:02,568 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.48 % 2021-12-02 06:35:02,576 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:35:02,576 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:35:04,223 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 58.48 % 2021-12-02 06:35:04,223 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 58.48 2021-12-02 06:35:04,247 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:35:04,250 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:35:04,250 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:35:12,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.62 % 2021-12-02 06:35:12,596 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:35:12,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:35:14,259 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.62 % 2021-12-02 06:35:14,260 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 59.62 2021-12-02 06:35:14,273 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:35:14,276 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:35:14,276 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:35:22,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.62 % 2021-12-02 06:35:22,617 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:35:22,617 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:35:24,286 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.62 % 2021-12-02 06:35:24,286 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 59.62 2021-12-02 06:35:24,302 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:35:24,303 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:35:24,303 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:35:32,627 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.62 % 2021-12-02 06:35:32,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:35:32,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:35:34,315 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.62 % 2021-12-02 06:35:34,315 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 59.62 2021-12-02 06:35:34,329 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:35:34,332 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:35:34,332 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:35:42,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.62 % 2021-12-02 06:35:42,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:35:42,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:35:44,345 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.62 % 2021-12-02 06:35:44,345 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 59.62 2021-12-02 06:35:44,365 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:35:44,368 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:35:44,368 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:35:52,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.62 % 2021-12-02 06:35:52,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:35:52,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:35:54,378 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.62 % 2021-12-02 06:35:54,378 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 59.62 2021-12-02 06:35:54,392 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:35:54,395 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:35:54,397 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:36:02,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.62 % 2021-12-02 06:36:02,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:36:02,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:36:04,407 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 59.62 % 2021-12-02 06:36:04,407 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 59.62 2021-12-02 06:36:04,423 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:36:04,426 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:36:04,426 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:36:12,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 60.32 % 2021-12-02 06:36:12,719 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:36:12,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:36:14,437 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 60.32 % 2021-12-02 06:36:14,438 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 60.32 2021-12-02 06:36:14,457 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:36:14,460 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:36:14,460 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:36:22,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 61.57 % 2021-12-02 06:36:22,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:36:22,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:36:24,471 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 61.57 % 2021-12-02 06:36:24,473 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 61.57 2021-12-02 06:36:24,489 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:36:24,492 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:36:24,492 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:36:32,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 62.82 % 2021-12-02 06:36:32,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:36:32,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:36:34,502 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 62.82 % 2021-12-02 06:36:34,502 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 62.82 2021-12-02 06:36:34,516 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:36:34,519 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:36:34,519 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:36:42,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 62.82 % 2021-12-02 06:36:42,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:36:42,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:36:44,529 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 62.82 % 2021-12-02 06:36:44,529 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 62.82 2021-12-02 06:36:44,542 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:36:44,546 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:36:44,546 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:36:52,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 62.82 % 2021-12-02 06:36:52,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:36:52,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:36:54,556 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 62.82 % 2021-12-02 06:36:54,556 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 62.82 2021-12-02 06:36:54,571 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:36:54,573 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:36:54,573 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:37:02,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 63.97 % 2021-12-02 06:37:02,816 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:37:02,816 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:37:04,588 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 63.97 % 2021-12-02 06:37:04,588 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 63.97 2021-12-02 06:37:04,605 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:37:04,608 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:37:04,608 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:37:12,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 63.97 % 2021-12-02 06:37:12,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:37:12,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:37:14,618 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 63.97 % 2021-12-02 06:37:14,618 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 63.97 2021-12-02 06:37:14,632 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:37:14,635 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:37:14,637 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:37:22,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 63.97 % 2021-12-02 06:37:22,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:37:22,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:37:24,647 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 63.97 % 2021-12-02 06:37:24,647 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 63.97 2021-12-02 06:37:24,661 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:37:24,664 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:37:24,664 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:37:32,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 63.97 % 2021-12-02 06:37:32,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:37:32,877 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:37:34,677 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 63.97 % 2021-12-02 06:37:34,677 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 63.97 2021-12-02 06:37:34,697 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:37:34,701 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:37:34,701 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:37:42,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 64.67 % 2021-12-02 06:37:42,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:37:42,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:37:44,716 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 64.67 % 2021-12-02 06:37:44,716 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 64.67 2021-12-02 06:37:44,740 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:37:44,744 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:37:44,746 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:37:52,905 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 64.67 % 2021-12-02 06:37:52,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:37:52,913 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:37:54,759 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 64.67 % 2021-12-02 06:37:54,759 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 64.67 2021-12-02 06:37:54,776 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:37:54,779 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:37:54,779 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:38:02,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 65.82 % 2021-12-02 06:38:02,928 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:38:02,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:38:04,790 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 65.82 % 2021-12-02 06:38:04,792 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 65.82 2021-12-02 06:38:04,808 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:38:04,809 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:38:04,809 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:38:12,938 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 65.82 % 2021-12-02 06:38:12,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:38:12,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:38:14,821 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 65.82 % 2021-12-02 06:38:14,821 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 65.82 2021-12-02 06:38:14,835 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:38:14,836 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:38:14,838 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:38:22,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 66.52 % 2021-12-02 06:38:22,961 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:38:22,961 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:38:24,851 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 66.52 % 2021-12-02 06:38:24,851 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 66.52 2021-12-02 06:38:24,865 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:38:24,868 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:38:24,868 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:38:32,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 66.52 % 2021-12-02 06:38:32,977 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:38:32,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:38:34,878 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 66.52 % 2021-12-02 06:38:34,878 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 66.52 2021-12-02 06:38:34,894 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:38:34,895 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:38:34,895 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:38:42,992 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 67.67 % 2021-12-02 06:38:42,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:38:42,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:38:44,907 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 67.67 % 2021-12-02 06:38:44,907 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 67.67 2021-12-02 06:38:44,927 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:38:44,930 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:38:44,930 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:38:53,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 67.67 % 2021-12-02 06:38:53,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:38:53,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:38:54,944 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 67.67 % 2021-12-02 06:38:54,944 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 67.67 2021-12-02 06:38:54,966 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:38:54,967 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:38:54,967 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:39:03,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 67.67 % 2021-12-02 06:39:03,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:39:03,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:39:04,977 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 67.67 % 2021-12-02 06:39:04,977 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 67.67 2021-12-02 06:39:04,993 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:39:04,996 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:39:04,996 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:39:13,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 67.67 % 2021-12-02 06:39:13,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:39:13,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:39:15,006 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 67.67 % 2021-12-02 06:39:15,006 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 67.67 2021-12-02 06:39:15,022 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:39:15,023 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:39:15,023 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:39:23,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 67.67 % 2021-12-02 06:39:23,063 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:39:23,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:39:25,036 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 68.37 % 2021-12-02 06:39:25,036 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 68.37 2021-12-02 06:39:25,051 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:39:25,052 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:39:25,052 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:39:33,073 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 68.37 % 2021-12-02 06:39:33,079 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:39:33,079 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:39:35,065 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 68.37 % 2021-12-02 06:39:35,065 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 68.37 2021-12-02 06:39:35,084 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:39:35,085 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:39:35,085 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:39:43,091 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 69.52 % 2021-12-02 06:39:43,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:39:43,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:39:45,099 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 69.52 % 2021-12-02 06:39:45,101 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 69.52 2021-12-02 06:39:45,121 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:39:45,122 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:39:45,122 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:39:53,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 69.52 % 2021-12-02 06:39:53,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:39:53,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:39:55,134 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 69.52 % 2021-12-02 06:39:55,134 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 69.52 2021-12-02 06:39:55,153 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:39:55,155 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:39:55,157 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:40:03,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 69.52 % 2021-12-02 06:40:03,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:40:03,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:40:05,167 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 69.52 % 2021-12-02 06:40:05,167 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 69.52 2021-12-02 06:40:05,183 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:40:05,186 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:40:05,186 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:40:13,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 70.22 % 2021-12-02 06:40:13,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:40:13,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:40:15,201 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 70.22 % 2021-12-02 06:40:15,201 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 70.22 2021-12-02 06:40:15,219 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:40:15,221 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:40:15,221 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:40:23,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 70.22 % 2021-12-02 06:40:23,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:40:23,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:40:25,233 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 71.37 % 2021-12-02 06:40:25,233 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 71.37 2021-12-02 06:40:25,247 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:40:25,250 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:40:25,250 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:40:33,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 71.37 % 2021-12-02 06:40:33,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:40:33,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:40:35,260 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 71.37 % 2021-12-02 06:40:35,262 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 71.37 2021-12-02 06:40:35,276 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:40:35,279 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:40:35,279 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:40:43,201 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 71.37 % 2021-12-02 06:40:43,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:40:43,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:40:45,289 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 71.37 % 2021-12-02 06:40:45,289 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 71.37 2021-12-02 06:40:45,305 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:40:45,308 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:40:45,308 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:40:53,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 71.37 % 2021-12-02 06:40:53,229 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:40:53,229 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:40:55,321 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 71.37 % 2021-12-02 06:40:55,321 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 71.37 2021-12-02 06:40:55,335 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:40:55,338 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:40:55,338 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:41:03,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 71.37 % 2021-12-02 06:41:03,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:41:03,249 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:41:05,348 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 71.37 % 2021-12-02 06:41:05,348 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 71.37 2021-12-02 06:41:05,365 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:41:05,367 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:41:05,367 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:41:13,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 72.07 % 2021-12-02 06:41:13,265 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:41:13,265 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:41:15,378 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 72.07 % 2021-12-02 06:41:15,378 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 72.07 2021-12-02 06:41:15,395 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:41:15,400 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:41:15,400 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:41:23,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 73.22 % 2021-12-02 06:41:23,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:41:23,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:41:25,411 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 73.22 % 2021-12-02 06:41:25,411 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 73.22 2021-12-02 06:41:25,426 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:41:25,428 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:41:25,428 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:41:33,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 73.22 % 2021-12-02 06:41:33,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:41:33,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:41:35,440 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 73.22 % 2021-12-02 06:41:35,440 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 73.22 2021-12-02 06:41:35,456 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:41:35,459 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:41:35,459 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:41:43,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 73.22 % 2021-12-02 06:41:43,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:41:43,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:41:45,467 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 73.22 % 2021-12-02 06:41:45,469 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 73.22 2021-12-02 06:41:45,483 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:41:45,486 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:41:45,486 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:41:53,354 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 73.92 % 2021-12-02 06:41:53,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:41:53,362 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:41:55,499 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 73.92 % 2021-12-02 06:41:55,500 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 73.92 2021-12-02 06:41:55,516 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:41:55,519 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:41:55,519 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:42:03,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 73.92 % 2021-12-02 06:42:03,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:42:03,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:42:05,529 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 73.92 % 2021-12-02 06:42:05,530 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 73.92 2021-12-02 06:42:05,546 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:42:05,548 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:42:05,548 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:42:13,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 75.06 % 2021-12-02 06:42:13,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:42:13,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:42:15,559 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 75.06 % 2021-12-02 06:42:15,559 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 75.06 2021-12-02 06:42:15,575 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:42:15,576 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:42:15,576 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:42:23,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 75.06 % 2021-12-02 06:42:23,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:42:23,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:42:25,586 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 75.06 % 2021-12-02 06:42:25,586 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 75.06 2021-12-02 06:42:25,601 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:42:25,602 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:42:25,602 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:42:33,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 75.06 % 2021-12-02 06:42:33,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:42:33,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:42:35,612 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 75.06 % 2021-12-02 06:42:35,614 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 75.06 2021-12-02 06:42:35,628 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:42:35,632 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:42:35,632 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:42:43,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 75.06 % 2021-12-02 06:42:43,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:42:43,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:42:45,642 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 75.06 % 2021-12-02 06:42:45,642 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 75.06 2021-12-02 06:42:45,658 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:42:45,663 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:42:45,663 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:42:53,502 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.21 % 2021-12-02 06:42:53,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:42:53,512 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:42:55,673 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.21 % 2021-12-02 06:42:55,673 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 76.21 2021-12-02 06:42:55,688 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:42:55,690 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:42:55,691 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:43:03,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.21 % 2021-12-02 06:43:03,530 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:43:03,530 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:43:05,701 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.21 % 2021-12-02 06:43:05,701 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 76.21 2021-12-02 06:43:05,714 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:43:05,717 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:43:05,717 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:43:13,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.21 % 2021-12-02 06:43:13,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:43:13,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:43:15,727 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.21 % 2021-12-02 06:43:15,729 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 76.21 2021-12-02 06:43:15,743 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:43:15,746 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:43:15,746 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:43:23,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.21 % 2021-12-02 06:43:23,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:43:23,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:43:25,759 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.21 % 2021-12-02 06:43:25,760 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 76.21 2021-12-02 06:43:25,789 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:43:25,792 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:43:25,792 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:43:33,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:43:33,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:43:33,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:43:35,805 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:43:35,805 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 76.91 2021-12-02 06:43:35,822 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:43:35,825 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:43:35,825 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:43:43,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:43:43,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:43:43,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:43:45,859 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:43:45,859 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 76.91 2021-12-02 06:43:45,874 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:43:45,875 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:43:45,877 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:43:53,619 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:43:53,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:43:53,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:43:55,888 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:43:55,888 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 76.91 2021-12-02 06:43:55,901 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:43:55,904 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:43:55,904 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:44:03,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:44:03,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:44:03,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:44:05,914 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:44:05,914 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 76.91 2021-12-02 06:44:05,934 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:44:05,936 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:44:05,936 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:44:13,660 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:44:13,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:44:13,668 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:44:15,947 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:44:15,947 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 76.91 2021-12-02 06:44:15,963 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:44:15,966 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:44:15,966 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:44:23,678 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:44:23,688 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:44:23,688 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:44:25,979 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:44:25,980 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 76.91 2021-12-02 06:44:26,003 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:44:26,005 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:44:26,005 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:44:33,698 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:44:33,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:44:33,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:44:36,016 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:44:36,016 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 76.91 2021-12-02 06:44:36,038 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:44:36,039 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:44:36,039 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:44:43,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:44:43,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:44:43,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:44:46,051 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:44:46,052 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 76.91 2021-12-02 06:44:46,075 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:44:46,076 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:44:46,076 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:44:53,737 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 76.91 % 2021-12-02 06:44:53,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:44:53,744 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:44:56,088 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 77.61 % 2021-12-02 06:44:56,088 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 77.61 2021-12-02 06:44:56,108 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:44:56,111 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:44:56,111 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:45:03,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 77.61 % 2021-12-02 06:45:03,765 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:45:03,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:45:06,121 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 77.61 % 2021-12-02 06:45:06,122 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 77.61 2021-12-02 06:45:06,141 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:45:06,144 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:45:06,144 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:45:13,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 77.61 % 2021-12-02 06:45:13,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:45:13,783 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:45:16,154 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 77.61 % 2021-12-02 06:45:16,154 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 77.61 2021-12-02 06:45:16,177 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:45:16,178 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:45:16,180 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:45:23,793 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 77.61 % 2021-12-02 06:45:23,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:45:23,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:45:26,190 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 78.31 % 2021-12-02 06:45:26,190 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 78.31 2021-12-02 06:45:26,210 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:45:26,211 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:45:26,211 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:45:33,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 78.31 % 2021-12-02 06:45:33,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:45:33,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:45:36,223 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 78.31 % 2021-12-02 06:45:36,223 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 78.31 2021-12-02 06:45:36,242 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:45:36,244 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:45:36,244 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:45:43,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 78.31 % 2021-12-02 06:45:43,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:45:43,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:45:46,256 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 78.31 % 2021-12-02 06:45:46,256 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 78.31 2021-12-02 06:45:46,272 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:45:46,275 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:45:46,275 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:45:53,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 78.31 % 2021-12-02 06:45:53,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:45:53,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:45:56,286 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 78.31 % 2021-12-02 06:45:56,286 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 78.31 2021-12-02 06:45:56,302 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:45:56,305 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:45:56,305 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:46:03,867 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 78.31 % 2021-12-02 06:46:03,872 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:46:03,872 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:46:06,315 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 78.31 % 2021-12-02 06:46:06,315 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 78.31 2021-12-02 06:46:06,331 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:46:06,332 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:46:06,332 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:46:13,882 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 78.31 % 2021-12-02 06:46:13,890 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:46:13,890 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:46:16,345 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 78.31 % 2021-12-02 06:46:16,345 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 78.31 2021-12-02 06:46:16,361 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:46:16,362 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:46:16,362 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:46:23,900 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.01 % 2021-12-02 06:46:23,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:46:23,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:46:26,372 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.01 % 2021-12-02 06:46:26,372 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 79.01 2021-12-02 06:46:26,388 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:46:26,391 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:46:26,391 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:46:33,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.01 % 2021-12-02 06:46:33,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:46:33,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:46:36,401 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.01 % 2021-12-02 06:46:36,401 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 79.01 2021-12-02 06:46:36,415 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:46:36,417 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:46:36,418 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:46:43,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.01 % 2021-12-02 06:46:43,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:46:43,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:46:46,428 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.01 % 2021-12-02 06:46:46,428 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 79.01 2021-12-02 06:46:46,443 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:46:46,446 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:46:46,446 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:46:53,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.01 % 2021-12-02 06:46:53,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:46:53,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:46:56,457 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:46:56,457 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 79.71 2021-12-02 06:46:56,476 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:46:56,479 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:46:56,479 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:47:03,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:47:03,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:47:03,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:47:06,490 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:47:06,490 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 79.71 2021-12-02 06:47:06,505 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:47:06,507 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:47:06,507 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:47:13,992 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:47:13,999 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:47:13,999 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:47:16,519 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:47:16,519 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 79.71 2021-12-02 06:47:16,536 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:47:16,539 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:47:16,539 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:47:24,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:47:24,015 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:47:24,015 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:47:26,549 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:47:26,549 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 79.71 2021-12-02 06:47:26,562 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:47:26,565 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:47:26,565 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:47:34,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:47:34,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:47:34,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:47:36,575 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:47:36,575 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 79.71 2021-12-02 06:47:36,589 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:47:36,592 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:47:36,592 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:47:44,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:47:44,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:47:44,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:47:46,605 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:47:46,605 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 79.71 2021-12-02 06:47:46,632 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:47:46,635 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:47:46,635 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:47:54,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:47:54,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:47:54,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:47:56,647 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:47:56,647 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 79.71 2021-12-02 06:47:56,661 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:47:56,664 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:47:56,664 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:48:04,079 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:48:04,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:48:04,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:48:06,677 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:48:06,677 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 79.71 2021-12-02 06:48:06,697 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:48:06,700 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:48:06,700 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:48:14,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 79.71 % 2021-12-02 06:48:14,104 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:48:14,104 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:48:16,713 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 80.41 % 2021-12-02 06:48:16,713 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 80.41 2021-12-02 06:48:16,730 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:48:16,732 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:48:16,733 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:48:24,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 80.41 % 2021-12-02 06:48:24,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 06:48:24,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:48:26,743 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 80.41 % 2021-12-02 06:48:26,743 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 80.41 2021-12-02 06:48:26,759 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:48:26,760 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:48:26,760 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:48:34,121 | test | INFO | MainThread | [rest_client:rebalance_reached:119] Rebalance reached >80.41% in 1863.57999992 seconds 2021-12-02 06:48:34,137 | test | DEBUG | MainThread | [rest_client:diag_eval:1453] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-02 06:48:34,138 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-02 06:48:34,138 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 06:48:34,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:48:34,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 06:48:34,552 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:48:34,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 06:48:34,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 06:48:34,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 06:48:34,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 06:48:34,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 06:48:34,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 06:48:34,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 06:48:34,663 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 06:48:34,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 06:48:34,720 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['291245 memcached\n'], []) 2021-12-02 06:48:34,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 06:48:34,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 06:48:34,815 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['291245 memcached\n'], []) 2021-12-02 06:48:34,815 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 06:48:34,819 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 06:48:34,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:48:35,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 06:48:35,181 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:48:35,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 06:48:35,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 06:48:35,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 06:48:35,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 06:48:35,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 06:48:35,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 06:48:35,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 06:48:35,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:48:35,292 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 06:48:35,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:48:35,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:35,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:35,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:48:35,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:48:35,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:35,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:35,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:48:35,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:48:35,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:35,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:35,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:48:35,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:48:35,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:35,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:35,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:48:35,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:48:35,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:48:35,743 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 06:48:35,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:48:35,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:35,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:48:35,779 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 06:48:35,780 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 06:48:35,783 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 06:48:35,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:48:36,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 06:48:36,279 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:48:36,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 06:48:36,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 06:48:36,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 06:48:36,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 06:48:36,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 06:48:36,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 06:48:36,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 06:48:36,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:48:36,428 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 06:48:36,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:48:36,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:36,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:36,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:48:36,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:48:36,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:48:36,581 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 06:48:36,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:48:36,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:36,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:48:36,619 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 06:48:36,619 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 06:48:36,622 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 06:48:36,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:48:36,773 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 80.41 % 2021-12-02 06:48:36,773 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 80.41 2021-12-02 06:48:36,792 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:48:36,795 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:48:36,795 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:48:36,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 06:48:37,069 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:48:37,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 06:48:37,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 06:48:37,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 06:48:37,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 06:48:37,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 06:48:37,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 06:48:37,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 06:48:37,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:48:37,230 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 06:48:37,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:48:37,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:37,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:37,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:48:37,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:48:37,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:37,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:37,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:48:37,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:48:37,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:37,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:37,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:48:37,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:48:37,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:37,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:37,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:48:37,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:48:37,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:37,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:37,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 06:48:37,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 06:48:37,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:37,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:37,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 06:48:37,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 06:48:37,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:48:37,973 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 06:48:37,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:48:37,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:38,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:48:38,029 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 06:48:38,030 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 06:48:38,033 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 06:48:38,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:48:38,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 06:48:38,486 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:48:38,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 06:48:38,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 06:48:38,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 06:48:38,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 06:48:38,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 06:48:38,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 06:48:38,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 06:48:38,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:48:38,614 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 06:48:38,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:48:38,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:38,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:38,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:48:38,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:48:38,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:38,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:38,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:48:38,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:48:38,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:39,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:48:39,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:48:39,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:48:39,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:48:39,138 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 06:48:39,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:48:39,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:48:39,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:48:39,190 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 06:48:39,191 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 06:48:39,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-02 06:48:39,198 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-02 06:48:46,806 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 80.41 % 2021-12-02 06:48:46,806 | test | INFO | pool-3-thread-23 | [task:check:384] Rebalance - status: running, progress: 80.41 2021-12-02 06:48:46,829 | infra | DEBUG | pool-3-thread-23 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:48:46,832 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:48:46,832 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:48:56,842 | test | ERROR | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1640] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'9b32dd199be24cf13ad9bec763ef12d4', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=288cb523470489e6bb3ff2047d52e9ad', u'status': u'notRunning'} - rebalance failed 2021-12-02 06:48:56,858 | test | INFO | pool-3-thread-23 | [rest_client:print_UI_logs:2789] Latest logs from UI on 172.23.110.65: 2021-12-02 06:48:56,859 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638456526912L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:48:46.912Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.28373.127>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",795,8200181,\n [{\'ns_1@172.23.110.64\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.22254.124>,\n {wait_seqno_persisted,795,8200181}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,506}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = 49fd9404c916809e4f392b97dde62be8'} 2021-12-02 06:48:56,859 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638456526909L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:48:46.909Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-02 06:48:56,859 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638456526908L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:48:46.908Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1202 06:09:18.626046 291475 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-02 06:48:56,859 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638456526882L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:48:46.882Z', u'text': u'Worker <0.18825.127> (for action {move,\n {795,\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.64\'],\n [\'ns_1@172.23.110.68\',\n \'ns_1@172.23.110.64\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.28373.127>,\n {{wait_seqno_persisted_failed,\n "GleamBookUsers0",\n 795,\n 8200181,\n [{\'ns_1@172.23.110.64\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.22254.124>,\n {wait_seqno_persisted,\n 795,\n 8200181}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 506}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 211}]}]}}}'} 2021-12-02 06:48:56,861 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638454530997L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:15:30.997Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-02 06:48:56,861 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638454530779L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:15:30.779Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-02 06:48:56,861 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638454530552L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:15:30.552Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.68',\n 'ns_1@172.23.110.66','ns_1@172.23.110.65'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 49fd9404c916809e4f392b97dde62be8"} 2021-12-02 06:48:56,861 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638454518196L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:15:18.196Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 341 seconds.'} 2021-12-02 06:48:56,861 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638454150050L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:09:10.050Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.31886.123>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,93,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.20098.118>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 93}},\n infinity]}}}}}.\nRebalance Operation Id = e66aab4525440aa2a2befd0068c07d29'} 2021-12-02 06:48:56,861 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638454150042L, u'shortText': u'message', u'serverTime': u'2021-12-02T06:09:10.042Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1202 05:05:41.473266 286178 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-02 06:48:56,861 | test | ERROR | pool-3-thread-23 | [task:call:296] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'9b32dd199be24cf13ad9bec763ef12d4', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=288cb523470489e6bb3ff2047d52e9ad', u'status': u'notRunning'} - rebalance failed 2021-12-02 06:49:39,213 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-02 06:49:39,213 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 06:49:39,224 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-02 06:49:39,237 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.68:8091 2021-12-02 06:49:39,237 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 06:49:39,250 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.68:8091 is running 2021-12-02 06:49:39,263 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-02 06:49:39,263 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 06:49:39,275 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:49:39,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:49:39,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:49:41,295 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:49:41,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:49:41,301 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:49:43,315 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:49:43,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:49:43,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:49:45,335 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:49:45,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:49:45,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:49:47,352 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:49:47,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:49:47,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:49:49,371 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:49:49,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:49:49,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:49:51,391 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:49:51,400 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:49:51,400 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:49:53,411 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:49:53,418 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:49:53,418 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:49:55,430 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:49:55,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:49:55,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:49:57,448 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:49:57,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:49:57,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:49:59,467 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:49:59,474 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:49:59,474 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:01,486 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:01,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:01,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:03,506 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:03,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:03,512 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:05,526 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:05,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:05,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:07,543 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:07,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:07,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:09,565 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:09,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:09,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:11,586 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:11,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:11,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:13,609 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:13,617 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:13,617 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:15,631 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:15,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:15,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:17,651 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:17,658 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:17,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:19,673 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:19,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:19,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:21,691 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:21,698 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:21,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:23,710 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:23,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:23,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:25,727 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:25,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:25,734 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:27,746 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:27,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:27,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:29,766 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:29,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:29,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:31,783 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:31,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:31,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:33,802 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:33,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:33,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:35,821 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:35,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:35,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:37,841 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:37,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:37,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:39,862 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:39,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:39,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:41,884 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:41,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:41,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:43,903 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:43,910 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:43,910 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:45,921 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:45,928 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:45,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:47,941 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:47,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:47,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:49,960 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:49,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:49,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:51,983 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:51,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:51,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:54,007 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:54,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:54,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:56,032 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:56,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:56,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:50:58,053 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:50:58,061 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:50:58,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:00,081 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:00,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:00,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:02,102 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:02,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:02,109 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:04,122 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:04,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:04,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:06,144 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:06,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:06,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:08,164 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:08,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:08,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:10,186 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:10,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:10,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:12,207 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:12,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:12,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:14,227 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:14,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:14,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:16,250 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:16,257 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:16,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:18,273 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:18,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:18,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:20,295 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:20,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:20,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:22,315 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:22,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:22,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:24,335 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:24,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:24,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:26,357 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:26,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:26,364 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:28,378 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:28,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:28,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:30,398 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:30,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:30,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:32,418 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:32,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:32,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:34,440 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:34,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:34,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:36,460 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:36,466 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:36,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:38,480 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:38,487 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:38,487 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:40,503 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:40,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:40,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:42,523 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:42,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:42,530 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:44,543 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:44,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:44,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:46,563 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:46,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:46,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:48,582 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:48,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:48,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:50,601 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:50,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:50,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:52,624 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:52,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:52,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:54,645 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:54,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:54,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:56,665 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:56,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:56,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:51:58,686 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:51:58,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:51:58,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:00,707 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:00,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:00,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:02,729 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:02,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:02,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:04,750 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:04,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:04,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:06,769 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:06,775 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:06,775 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:08,788 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:08,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:08,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:10,805 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:10,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:10,813 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:12,826 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:12,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:12,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:14,844 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:14,849 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:14,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:16,864 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:16,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:16,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:18,884 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:18,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:18,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:20,904 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:20,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:20,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:22,924 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:22,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:22,931 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:24,943 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:24,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:24,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:26,960 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:26,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:26,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:28,980 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:28,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:28,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:31,000 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:31,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:31,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:33,023 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:33,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:33,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:35,039 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:35,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:35,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:37,061 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:37,066 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:37,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:39,081 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:39,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:39,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:41,099 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:41,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:41,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:43,119 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:43,127 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:43,127 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:45,138 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:45,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:45,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:47,161 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:47,168 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:47,170 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:49,184 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:49,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:49,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:51,201 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:51,209 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:51,209 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:53,223 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:53,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:53,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:55,243 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:55,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:55,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:57,263 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:57,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:57,270 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:52:59,285 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:52:59,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:52:59,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:01,305 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:01,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:01,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:03,325 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:03,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:03,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:05,344 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:05,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:05,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:07,361 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:07,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:07,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:09,378 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:09,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:09,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:11,395 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:11,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:11,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:13,415 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:13,423 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:13,423 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:15,437 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:15,443 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:15,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:17,457 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:17,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:17,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:19,476 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:19,482 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:19,482 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:21,493 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:21,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:21,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:23,512 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:23,517 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:23,517 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:25,533 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:25,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:25,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:27,555 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:27,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:27,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:29,573 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:29,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:29,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:31,592 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:31,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:31,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:33,611 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:33,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:33,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:35,631 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:35,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:35,640 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:37,651 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:37,660 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:37,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:39,671 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:39,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:39,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:41,688 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:41,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:41,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:43,713 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:43,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:43,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:45,733 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:45,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:45,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:47,752 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:47,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:47,762 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:49,773 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:49,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:49,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:51,790 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:51,798 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:51,798 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:53,809 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:53,816 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:53,816 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:55,831 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:55,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:55,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:57,851 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:57,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:57,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:53:59,872 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:53:59,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:53:59,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:01,897 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:01,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:01,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:03,918 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:03,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:03,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:05,941 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:05,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:05,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:07,964 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:07,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:07,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:09,990 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:09,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:09,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:12,013 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:12,022 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:12,022 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:14,036 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:14,043 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:14,045 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:16,061 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:16,066 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:16,066 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:18,082 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:18,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:18,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:20,109 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:20,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:20,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:22,137 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:22,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:22,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:24,157 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:24,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:24,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:26,177 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:26,186 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:26,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:28,198 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:28,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:28,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:30,220 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:30,227 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:30,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:32,242 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:32,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:32,247 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:34,263 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:34,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:34,270 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:36,285 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:36,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:36,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:38,305 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:38,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:38,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:40,325 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:40,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:40,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:42,346 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:42,354 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:42,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:44,368 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:44,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:44,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:46,388 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:46,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:46,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:48,408 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:48,415 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:48,415 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:50,428 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:50,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:50,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:52,450 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:52,459 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:52,459 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:54,473 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:54,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:54,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:56,493 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:56,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:56,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:54:58,516 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 06:54:58,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 06:54:58,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 06:55:00,536 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-02 06:55:00,546 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-02 06:55:00,546 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 06:55:00,558 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-02 06:55:00,559 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 06:55:00,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:55:00,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 06:55:00,880 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:55:00,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 06:55:00,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 06:55:00,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 06:55:00,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 06:55:00,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 06:55:00,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 06:55:00,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 06:55:00,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:55:00,973 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 06:55:01,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:55:01,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:01,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:01,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:55:01,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:55:01,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:01,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:01,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:55:01,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:55:01,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:01,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:01,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:55:01,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:55:01,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:01,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:01,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:55:01,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:55:01,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:55:01,496 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 06:55:01,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:55:01,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:01,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:55:01,535 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 06:55:01,535 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 06:55:01,538 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 06:55:01,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:55:01,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 06:55:01,878 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:55:01,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 06:55:01,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 06:55:01,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 06:55:01,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 06:55:01,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 06:55:01,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 06:55:01,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 06:55:01,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:55:01,966 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 06:55:01,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:55:02,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:02,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:02,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:55:02,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:55:02,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:55:02,130 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 06:55:02,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:55:02,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:02,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:55:02,171 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 06:55:02,171 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 06:55:02,176 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 06:55:02,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:55:02,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 06:55:02,608 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:55:02,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 06:55:02,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 06:55:02,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 06:55:02,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 06:55:02,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 06:55:02,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 06:55:02,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 06:55:02,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:55:02,733 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 06:55:02,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:55:02,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:02,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:02,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:55:02,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:55:02,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:02,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:02,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:55:02,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:55:03,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:03,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:03,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:55:03,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:55:03,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:03,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:03,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:55:03,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 06:55:03,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:03,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:03,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 06:55:03,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 06:55:03,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:03,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:03,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 06:55:03,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 06:55:03,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:03,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:03,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 06:55:03,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 06:55:03,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:55:03,624 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 06:55:03,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:55:03,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:03,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:55:03,727 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 06:55:03,727 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 06:55:03,732 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 06:55:03,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 06:55:04,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 06:55:04,089 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 06:55:04,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 06:55:04,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 06:55:04,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 06:55:04,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 06:55:04,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 06:55:04,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 06:55:04,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 06:55:04,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 06:55:04,191 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 06:55:04,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 06:55:04,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:04,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:04,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:55:04,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 06:55:04,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:04,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:04,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:55:04,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 06:55:04,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:04,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 06:55:04,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:55:04,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 06:55:04,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 06:55:04,589 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 06:55:04,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 06:55:04,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 06:55:04,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 06:55:04,630 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 06:55:04,630 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 06:55:04,630 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1638454530.5 2021-12-02 06:55:04,631 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 80 2021-12-02 06:55:04,658 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1638456904.63 2021-12-02 06:55:04,661 | infra | INFO | pool-3-thread-24 | [task:start_task:89] Thread 'Rebalance_task_IN=[]_OUT=[]_1638456904.63' started 2021-12-02 06:55:04,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-02 06:55:04,673 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-02 06:55:04,686 | test | DEBUG | pool-3-thread-24 | [rest_client:rebalance:1431] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-02 06:55:04,740 | test | DEBUG | pool-3-thread-24 | [rest_client:rebalance:1436] Rebalance operation started 2021-12-02 06:55:04,743 | test | INFO | pool-3-thread-24 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1817-enterprise | 3.96675113475 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1817-enterprise | 0.889259413645 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1817-enterprise | 2.7641688008 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1817-enterprise | 2.72884139044 | Cluster node | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-02 06:55:04,753 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:55:04,753 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:55:04,769 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:55:04,772 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:55:04,772 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:55:14,782 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:55:14,782 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:55:14,798 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:55:14,801 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:55:14,801 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:55:24,812 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:55:24,812 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:55:24,828 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:55:24,831 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:55:24,831 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:55:34,844 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:55:34,844 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:55:34,862 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:55:34,867 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:55:34,867 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:55:44,878 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:55:44,878 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:55:44,892 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:55:44,894 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:55:44,894 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:55:54,904 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:55:54,905 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:55:54,918 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:55:54,921 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:55:54,921 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:56:04,930 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:56:04,930 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:56:04,947 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:56:04,950 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:56:04,950 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:56:14,961 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:56:14,963 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:56:14,979 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:56:14,980 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:56:14,980 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:56:24,992 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:56:24,992 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:56:25,012 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:56:25,015 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:56:25,015 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:56:35,026 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:56:35,026 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:56:35,042 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:56:35,046 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:56:35,046 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:56:45,058 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:56:45,058 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:56:45,082 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:56:45,085 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:56:45,085 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:56:55,095 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:56:55,096 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:56:55,112 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:56:55,115 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:56:55,115 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:57:04,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:57:04,684 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 0.0 2021-12-02 06:57:04,684 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1638456904.63 2021-12-02 06:57:05,131 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:57:05,131 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:57:05,148 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:57:05,151 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:57:05,151 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:57:15,161 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:57:15,163 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:57:15,178 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:57:15,180 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:57:15,181 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:57:25,194 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:57:25,196 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:57:25,221 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:57:25,224 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:57:25,224 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:57:35,237 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:57:35,237 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:57:35,262 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:57:35,263 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:57:35,263 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:57:45,278 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:57:45,278 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:57:45,299 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:57:45,302 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:57:45,302 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:57:55,313 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:57:55,313 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:57:55,332 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:57:55,335 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:57:55,336 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:58:05,351 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:58:05,351 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:58:05,365 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:58:05,368 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:58:05,368 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:58:15,378 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:58:15,380 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:58:15,398 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:58:15,401 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:58:15,401 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:58:25,411 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 06:58:25,411 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 06:58:25,427 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:58:25,430 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:58:25,430 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:58:35,440 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.57 % 2021-12-02 06:58:35,440 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 3.57 2021-12-02 06:58:35,454 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:58:35,457 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:58:35,457 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:58:45,467 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 06:58:45,467 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 06:58:45,483 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:58:45,486 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:58:45,486 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:58:55,496 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 06:58:55,496 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 06:58:55,513 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:58:55,515 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:58:55,516 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:59:05,529 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 06:59:05,529 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 06:59:05,546 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:59:05,549 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:59:05,549 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:59:15,566 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 06:59:15,566 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 06:59:15,584 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:59:15,586 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:59:15,586 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:59:25,598 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 06:59:25,598 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 06:59:25,615 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:59:25,618 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:59:25,618 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:59:35,628 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 06:59:35,628 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 06:59:35,644 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:59:35,647 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:59:35,647 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:59:45,658 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 06:59:45,658 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 06:59:45,674 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:59:45,677 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:59:45,677 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 06:59:55,687 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 06:59:55,687 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 06:59:55,703 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 06:59:55,704 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 06:59:55,706 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:00:05,717 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 07:00:05,717 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 07:00:05,733 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:00:05,736 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:00:05,736 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:00:15,746 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 07:00:15,747 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 07:00:15,762 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:00:15,766 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:00:15,766 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:00:25,779 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 07:00:25,779 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 07:00:25,796 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:00:25,799 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:00:25,799 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:00:35,809 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 07:00:35,809 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 07:00:35,826 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:00:35,831 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:00:35,831 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:00:45,842 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 07:00:45,842 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 07:00:45,858 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:00:45,862 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:00:45,862 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:00:55,871 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 07:00:55,872 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 07:00:55,887 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:00:55,888 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:00:55,888 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:01:05,901 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 07:01:05,901 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 07:01:05,917 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:01:05,920 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:01:05,920 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:01:15,930 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 07:01:15,930 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 07:01:15,944 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:01:15,946 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:01:15,946 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:01:25,956 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 07:01:25,956 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 07:01:25,970 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:01:25,971 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:01:25,971 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:01:35,983 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 07:01:35,983 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 07:01:35,999 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:01:36,000 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:01:36,000 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:01:46,012 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.14 % 2021-12-02 07:01:46,013 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 7.14 2021-12-02 07:01:46,028 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:01:46,029 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:01:46,029 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:01:56,040 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.71 % 2021-12-02 07:01:56,040 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 10.71 2021-12-02 07:01:56,058 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:01:56,059 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:01:56,059 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:02:06,069 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:02:06,071 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:02:06,084 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:02:06,085 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:02:06,085 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:02:16,109 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:02:16,109 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:02:16,130 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:02:16,131 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:02:16,131 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:02:26,142 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:02:26,142 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:02:26,161 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:02:26,163 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:02:26,164 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:02:36,174 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:02:36,174 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:02:36,194 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:02:36,197 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:02:36,197 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:02:46,207 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:02:46,207 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:02:46,223 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:02:46,224 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:02:46,224 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:02:56,236 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:02:56,236 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:02:56,253 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:02:56,255 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:02:56,255 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:03:06,266 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:03:06,266 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:03:06,286 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:03:06,288 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:03:06,288 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:03:16,299 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:03:16,301 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:03:16,319 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:03:16,322 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:03:16,322 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:03:26,335 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:03:26,335 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:03:26,348 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:03:26,351 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:03:26,351 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:03:36,361 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:03:36,362 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:03:36,381 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:03:36,384 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:03:36,384 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:03:46,398 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:03:46,398 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:03:46,421 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:03:46,424 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:03:46,424 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:03:56,437 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:03:56,437 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:03:56,460 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:03:56,463 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:03:56,463 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:04:06,476 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:04:06,476 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:04:06,496 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:04:06,499 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:04:06,499 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:04:16,509 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:04:16,509 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:04:16,528 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:04:16,530 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:04:16,530 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:04:26,540 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:04:26,540 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:04:26,555 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:04:26,559 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:04:26,559 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:04:36,569 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:04:36,569 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:04:36,585 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:04:36,588 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:04:36,588 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:04:46,598 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:04:46,598 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:04:46,628 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:04:46,631 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:04:46,631 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:04:56,641 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.29 % 2021-12-02 07:04:56,641 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 14.29 2021-12-02 07:04:56,657 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:04:56,660 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:04:56,660 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:05:06,670 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.07 % 2021-12-02 07:05:06,671 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 16.07 2021-12-02 07:05:06,688 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:05:06,691 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:05:06,691 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:05:16,707 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.86 % 2021-12-02 07:05:16,709 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 17.86 2021-12-02 07:05:16,723 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:05:16,726 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:05:16,726 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:05:26,736 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:05:26,737 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:05:26,756 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:05:26,759 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:05:26,759 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:05:36,769 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:05:36,769 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:05:36,785 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:05:36,788 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:05:36,788 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:05:46,801 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:05:46,801 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:05:46,815 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:05:46,819 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:05:46,819 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:05:56,839 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:05:56,839 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:05:56,854 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:05:56,857 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:05:56,857 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:06:06,868 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:06:06,868 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:06:06,884 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:06:06,887 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:06:06,887 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:06:16,897 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:06:16,898 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:06:16,911 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:06:16,914 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:06:16,914 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:06:26,926 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:06:26,926 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:06:26,940 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:06:26,943 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:06:26,943 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:06:36,953 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:06:36,953 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:06:36,970 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:06:36,971 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:06:36,971 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:06:46,982 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:06:46,983 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:06:46,996 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:06:47,000 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:06:47,000 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:06:57,010 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:06:57,010 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:06:57,025 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:06:57,026 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:06:57,026 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:07:07,039 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:07:07,039 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:07:07,053 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:07:07,055 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:07:07,055 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:07:17,073 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:07:17,073 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:07:17,091 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:07:17,092 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:07:17,092 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:07:27,104 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:07:27,105 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:07:27,117 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:07:27,118 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:07:27,118 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:07:37,128 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:07:37,128 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:07:37,141 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:07:37,144 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:07:37,144 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:07:47,154 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:07:47,154 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:07:47,171 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:07:47,174 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:07:47,174 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:07:57,184 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:07:57,184 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:07:57,198 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:07:57,200 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:07:57,200 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:08:07,211 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 21.43 % 2021-12-02 07:08:07,211 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 21.43 2021-12-02 07:08:07,230 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:08:07,233 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:08:07,233 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:08:17,243 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 23.21 % 2021-12-02 07:08:17,243 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 23.21 2021-12-02 07:08:17,263 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:08:17,266 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:08:17,266 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:08:27,276 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 25.00 % 2021-12-02 07:08:27,276 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 25.0 2021-12-02 07:08:27,299 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:08:27,302 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:08:27,302 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:08:37,313 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:08:37,313 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:08:37,329 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:08:37,332 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:08:37,332 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:08:47,344 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:08:47,344 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:08:47,364 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:08:47,365 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:08:47,365 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:08:57,378 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:08:57,378 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:08:57,392 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:08:57,394 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:08:57,394 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:09:07,405 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:09:07,405 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:09:07,420 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:09:07,423 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:09:07,423 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:09:17,434 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:09:17,434 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:09:17,454 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:09:17,457 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:09:17,457 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:09:27,467 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:09:27,467 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:09:27,483 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:09:27,486 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:09:27,486 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:09:37,497 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:09:37,499 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:09:37,516 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:09:37,519 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:09:37,519 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:09:47,532 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:09:47,532 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:09:47,551 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:09:47,555 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:09:47,555 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:09:57,566 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:09:57,566 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:09:57,598 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:09:57,601 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:09:57,601 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:10:07,618 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:10:07,619 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:10:07,634 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:10:07,638 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:10:07,638 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:10:17,651 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:10:17,651 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:10:17,668 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:10:17,671 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:10:17,671 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:10:27,681 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:10:27,683 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:10:27,697 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:10:27,701 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:10:27,701 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:10:37,713 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:10:37,713 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:10:37,727 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:10:37,732 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:10:37,732 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:10:47,743 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:10:47,743 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:10:47,756 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:10:47,759 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:10:47,759 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:10:57,770 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:10:57,772 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:10:57,789 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:10:57,792 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:10:57,792 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:11:07,803 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:11:07,805 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:11:07,821 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:11:07,823 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:11:07,823 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:11:17,835 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:11:17,835 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:11:17,851 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:11:17,855 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:11:17,855 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:11:27,867 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 28.57 % 2021-12-02 07:11:27,868 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 28.57 2021-12-02 07:11:27,881 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:11:27,884 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:11:27,884 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:11:37,895 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 32.14 % 2021-12-02 07:11:37,895 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 32.14 2021-12-02 07:11:37,913 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:11:37,915 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:11:37,917 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:11:47,927 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:11:47,927 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:11:47,943 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:11:47,946 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:11:47,946 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:11:57,957 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:11:57,957 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:11:57,971 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:11:57,974 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:11:57,976 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:12:07,987 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:12:07,989 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:12:08,003 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:12:08,006 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:12:08,006 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:12:18,020 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:12:18,022 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:12:18,036 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:12:18,039 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:12:18,039 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:12:28,049 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:12:28,051 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:12:28,065 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:12:28,066 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:12:28,066 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:12:38,076 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:12:38,076 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:12:38,091 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:12:38,092 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:12:38,092 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:12:48,102 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:12:48,104 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:12:48,117 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:12:48,118 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:12:48,118 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:12:58,130 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:12:58,130 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:12:58,142 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:12:58,144 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:12:58,144 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:13:08,157 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:13:08,157 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:13:08,174 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:13:08,176 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:13:08,176 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:13:18,187 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:13:18,187 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:13:18,200 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:13:18,201 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:13:18,203 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:13:28,213 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:13:28,213 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:13:28,234 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:13:28,236 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:13:28,236 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:13:38,246 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:13:38,247 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:13:38,262 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:13:38,263 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:13:38,265 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:13:48,276 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:13:48,276 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:13:48,296 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:13:48,299 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:13:48,299 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:13:58,311 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:13:58,312 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:13:58,331 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:13:58,332 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:13:58,334 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:14:08,344 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:14:08,344 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:14:08,357 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:14:08,358 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:14:08,359 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:14:18,371 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:14:18,372 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:14:18,391 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:14:18,394 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:14:18,394 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:14:28,407 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:14:28,407 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:14:28,427 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:14:28,428 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:14:28,428 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:14:38,440 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 35.71 % 2021-12-02 07:14:38,440 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 35.71 2021-12-02 07:14:38,461 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:14:38,464 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:14:38,464 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:14:48,480 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 37.50 % 2021-12-02 07:14:48,480 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 37.5 2021-12-02 07:14:48,505 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:14:48,507 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:14:48,507 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:14:58,519 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 39.29 % 2021-12-02 07:14:58,519 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 39.29 2021-12-02 07:14:58,533 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:14:58,536 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:14:58,536 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:15:08,546 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:15:08,546 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:15:08,565 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:15:08,566 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:15:08,568 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:15:18,578 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:15:18,578 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:15:18,592 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:15:18,595 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:15:18,595 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:15:28,605 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:15:28,605 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:15:28,621 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:15:28,622 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:15:28,624 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:15:38,634 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:15:38,634 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:15:38,648 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:15:38,651 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:15:38,651 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:15:48,661 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:15:48,661 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:15:48,676 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:15:48,677 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:15:48,677 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:15:58,688 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:15:58,688 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:15:58,703 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:15:58,707 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:15:58,707 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:16:08,717 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:16:08,717 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:16:08,733 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:16:08,736 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:16:08,736 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:16:18,746 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:16:18,746 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:16:18,762 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:16:18,766 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:16:18,766 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:16:28,778 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:16:28,778 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:16:28,793 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:16:28,796 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:16:28,796 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:16:38,809 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:16:38,809 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:16:38,825 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:16:38,828 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:16:38,828 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:16:48,839 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:16:48,841 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:16:48,857 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:16:48,861 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:16:48,861 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:16:58,884 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:16:58,884 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:16:58,901 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:16:58,904 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:16:58,904 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:17:08,915 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:17:08,915 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:17:08,930 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:17:08,931 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:17:08,931 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:17:18,944 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:17:18,944 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:17:18,961 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:17:18,964 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:17:18,964 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:17:28,974 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:17:28,974 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:17:28,993 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:17:28,996 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:17:28,996 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:17:39,006 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:17:39,006 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:17:39,022 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:17:39,025 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:17:39,025 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:17:49,036 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:17:49,036 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:17:49,052 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:17:49,055 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:17:49,055 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:17:59,069 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:17:59,069 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:17:59,082 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:17:59,085 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:17:59,085 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:18:09,096 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:18:09,098 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:18:09,112 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:18:09,115 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:18:09,115 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:18:19,125 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 42.86 % 2021-12-02 07:18:19,125 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 42.86 2021-12-02 07:18:19,138 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:18:19,141 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:18:19,141 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:18:29,151 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 46.43 % 2021-12-02 07:18:29,151 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: running, progress: 46.43 2021-12-02 07:18:29,164 | infra | DEBUG | pool-3-thread-24 | [task:check:411] Wait before next rebalance progress check 2021-12-02 07:18:29,167 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 07:18:29,167 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 07:18:39,180 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1649 2021-12-02 07:18:39,180 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-02 07:18:44,190 | test | INFO | pool-3-thread-24 | [task:check:384] Rebalance - status: none, progress: 100 2021-12-02 07:18:44,206 | test | INFO | pool-3-thread-24 | [task:check:443] Rebalance completed with progress: 100% in 1419.46600008 sec 2021-12-02 07:18:44,206 | infra | INFO | pool-3-thread-24 | [task:complete_task:104] Thread 'Rebalance_task_IN=[]_OUT=[]_1638456904.63' completed 2021-12-02 07:18:44,236 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3946] Stats: {u'opsPerSec': 50093, u'diskFetches': 235665006, u'dataUsed': 7930233343737L, u'itemCount': 3999988400L, u'quotaPercentUsed': 75.19209670647979, u'storageTotals': {u'hdd': {u'usedByData': 12585989139568L, u'used': 12824667509222L, u'quotaTotal': 47967746654208L, u'total': 47967746654208L, u'free': 34106304259692L}, u'ram': {u'usedByData': 322947596272L, u'used': 915888472064L, u'quotaTotal': 776848015360L, u'total': 1079482892288L, u'quotaUsed': 429496729600L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 12585989139568L, u'vbActiveNumNonResident': 3835739766L, u'memUsed': 322947596272L} 2021-12-02 07:18:44,512 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 3999988400 | 400.00 GiB | 300.77 GiB | 11.45 TiB | 4.10915744488 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ 2021-12-02 07:18:44,512 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+--------------------+--------------------+-------------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+--------------------+--------------------+-------------------+---------------+ | 4000000000 | 38000000000 | 80000000-800000000 | 80000000-800000000 | 40000000-80000000 | 0-0 | +---------------+---------------+--------------------+--------------------+-------------------+---------------+ 2021-12-02 07:18:44,773 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 4.10915744488 2021-12-02 07:18:44,776 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:18:44,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:18:45,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:18:45,122 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:18:45,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:18:45,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:18:45,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:18:45,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:18:45,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:18:45,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:18:45,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:18:45,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-02 07:18:45,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-02 07:18:45,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-02 07:18:45,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-02 07:18:46,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-02 07:18:48,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-936 is 366 2021-12-02 07:18:48,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-864 is 305 2021-12-02 07:18:48,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-72 is 355 2021-12-02 07:18:48,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-0 is 323 2021-12-02 07:18:48,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-1009 is 331 2021-12-02 07:18:48,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-937 is 311 2021-12-02 07:18:48,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-865 is 364 2021-12-02 07:18:48,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-577 is 348 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-73 is 371 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-1 is 351 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-82 is 359 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-10 is 362 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-515 is 338 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-11 is 355 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-84 is 386 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-12 is 331 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-85 is 307 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-13 is 353 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-374 is 311 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-951 is 308 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-159 is 300 2021-12-02 07:18:48,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-520 is 333 2021-12-02 07:18:48,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-88 is 305 2021-12-02 07:18:48,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-17 is 310 2021-12-02 07:18:48,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-18 is 323 2021-12-02 07:18:48,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-362 is 308 2021-12-02 07:18:48,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-146 is 304 2021-12-02 07:18:48,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-74 is 325 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-2 is 357 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-164 is 334 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-20 is 381 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-165 is 327 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-21 is 304 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-382 is 312 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-23 is 379 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-168 is 305 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.25/kvstore-169 is 300 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.26/kvstore-890 is 313 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-28 is 356 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-579 is 302 2021-12-02 07:18:48,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-75 is 353 2021-12-02 07:18:48,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-894 is 300 2021-12-02 07:18:48,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-390 is 311 2021-12-02 07:18:48,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-968 is 412 2021-12-02 07:18:48,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-896 is 367 2021-12-02 07:18:48,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-32 is 369 2021-12-02 07:18:48,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-538 is 342 2021-12-02 07:18:48,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.35/kvstore-395 is 337 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-972 is 304 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-36 is 326 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.37/kvstore-37 is 392 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-542 is 350 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-38 is 304 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-868 is 317 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-76 is 343 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-4 is 355 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-40 is 362 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.41/kvstore-41 is 348 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.42/kvstore-42 is 346 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-43 is 353 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-548 is 340 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-44 is 350 2021-12-02 07:18:48,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-45 is 312 2021-12-02 07:18:48,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-47 is 389 2021-12-02 07:18:48,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-408 is 333 2021-12-02 07:18:48,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-48 is 368 2021-12-02 07:18:48,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-409 is 302 2021-12-02 07:18:48,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-49 is 303 2021-12-02 07:18:48,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-149 is 315 2021-12-02 07:18:48,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-77 is 357 2021-12-02 07:18:48,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-5 is 409 2021-12-02 07:18:48,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-51 is 329 2021-12-02 07:18:48,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-988 is 301 2021-12-02 07:18:48,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-52 is 312 2021-12-02 07:18:48,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-53 is 350 2021-12-02 07:18:48,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-414 is 303 2021-12-02 07:18:48,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-342 is 311 2021-12-02 07:18:48,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-126 is 349 2021-12-02 07:18:48,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-415 is 307 2021-12-02 07:18:48,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-55 is 304 2021-12-02 07:18:48,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.56/kvstore-992 is 307 2021-12-02 07:18:48,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.56/kvstore-920 is 320 2021-12-02 07:18:48,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.56/kvstore-560 is 300 2021-12-02 07:18:48,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.56/kvstore-56 is 350 2021-12-02 07:18:48,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-561 is 359 2021-12-02 07:18:48,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-922 is 302 2021-12-02 07:18:48,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-418 is 329 2021-12-02 07:18:48,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-130 is 312 2021-12-02 07:18:48,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-923 is 302 2021-12-02 07:18:48,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-59 is 357 2021-12-02 07:18:48,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-870 is 318 2021-12-02 07:18:48,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-150 is 306 2021-12-02 07:18:48,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-78 is 367 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-925 is 300 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-422 is 359 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-62 is 327 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-423 is 388 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-351 is 311 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-63 is 394 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.64/kvstore-856 is 306 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-425 is 321 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-65 is 386 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-930 is 310 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-570 is 367 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-66 is 343 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-932 is 306 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-933 is 323 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-69 is 302 2021-12-02 07:18:48,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-1015 is 327 2021-12-02 07:18:48,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-151 is 333 2021-12-02 07:18:48,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-79 is 341 2021-12-02 07:18:48,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-7 is 376 2021-12-02 07:18:48,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.71/kvstore-143 is 345 2021-12-02 07:18:48,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-872 is 334 2021-12-02 07:18:48,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-152 is 400 2021-12-02 07:18:48,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-8 is 334 2021-12-02 07:18:48,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-1017 is 365 2021-12-02 07:18:48,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-513 is 301 2021-12-02 07:18:48,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-81 is 330 2021-12-02 07:18:48,598 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:18:48,599 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:18:48,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:18:48,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:18:48,990 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:18:48,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:18:49,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:18:49,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:18:49,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:18:49,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:18:49,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:18:49,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:18:49,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-02 07:18:49,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-02 07:18:49,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-02 07:18:49,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-02 07:18:49,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-02 07:18:51,533 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.0/kvstore-504 is 372 2021-12-02 07:18:51,535 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.0/kvstore-648 is 315 2021-12-02 07:18:51,535 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.1/kvstore-505 is 366 2021-12-02 07:18:51,535 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.11/kvstore-155 is 313 2021-12-02 07:18:51,535 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.11/kvstore-443 is 312 2021-12-02 07:18:51,535 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.15/kvstore-447 is 306 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.2/kvstore-506 is 341 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.21/kvstore-453 is 318 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.24/kvstore-456 is 319 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.28/kvstore-460 is 310 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.3/kvstore-507 is 329 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.30/kvstore-318 is 312 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.34/kvstore-466 is 307 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.4/kvstore-508 is 366 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.40/kvstore-472 is 336 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.41/kvstore-473 is 332 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.42/kvstore-474 is 352 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.5/kvstore-509 is 361 2021-12-02 07:18:51,536 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.55/kvstore-991 is 342 2021-12-02 07:18:51,538 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.55/kvstore-487 is 315 2021-12-02 07:18:51,538 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.58/kvstore-490 is 329 2021-12-02 07:18:51,538 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.59/kvstore-491 is 301 2021-12-02 07:18:51,538 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.6/kvstore-510 is 340 2021-12-02 07:18:51,538 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.60/kvstore-492 is 362 2021-12-02 07:18:51,538 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.63/kvstore-495 is 319 2021-12-02 07:18:51,538 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.64/kvstore-496 is 303 2021-12-02 07:18:51,539 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.66/kvstore-498 is 335 2021-12-02 07:18:51,539 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.69/kvstore-501 is 352 2021-12-02 07:18:51,539 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.70/kvstore-502 is 309 2021-12-02 07:18:51,539 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:18:51,542 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:18:51,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:18:51,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:18:51,884 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:18:51,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:18:51,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:18:51,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:18:51,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:18:51,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:18:51,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:18:52,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:18:52,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-02 07:18:52,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-02 07:18:52,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-02 07:18:52,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-02 07:18:52,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-02 07:18:56,315 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-792 is 425 2021-12-02 07:18:56,316 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-720 is 350 2021-12-02 07:18:56,316 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-504 is 425 2021-12-02 07:18:56,316 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-432 is 404 2021-12-02 07:18:56,316 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-360 is 414 2021-12-02 07:18:56,318 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-216 is 385 2021-12-02 07:18:56,318 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-72 is 438 2021-12-02 07:18:56,318 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-0 is 403 2021-12-02 07:18:56,318 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-793 is 386 2021-12-02 07:18:56,318 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-721 is 329 2021-12-02 07:18:56,318 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-505 is 408 2021-12-02 07:18:56,318 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-433 is 430 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-361 is 393 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-217 is 439 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-73 is 383 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-1 is 376 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-802 is 382 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-730 is 414 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-442 is 309 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-370 is 367 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-226 is 386 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-82 is 327 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-10 is 376 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-803 is 348 2021-12-02 07:18:56,319 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-731 is 412 2021-12-02 07:18:56,321 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-443 is 353 2021-12-02 07:18:56,321 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-371 is 347 2021-12-02 07:18:56,321 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-227 is 362 2021-12-02 07:18:56,321 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-83 is 321 2021-12-02 07:18:56,321 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-11 is 349 2021-12-02 07:18:56,322 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-804 is 394 2021-12-02 07:18:56,322 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-732 is 352 2021-12-02 07:18:56,322 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-444 is 393 2021-12-02 07:18:56,322 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-372 is 370 2021-12-02 07:18:56,322 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-228 is 392 2021-12-02 07:18:56,322 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-84 is 386 2021-12-02 07:18:56,322 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-12 is 406 2021-12-02 07:18:56,322 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-805 is 378 2021-12-02 07:18:56,322 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-733 is 356 2021-12-02 07:18:56,322 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-445 is 393 2021-12-02 07:18:56,322 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-373 is 363 2021-12-02 07:18:56,322 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-229 is 400 2021-12-02 07:18:56,323 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-85 is 344 2021-12-02 07:18:56,323 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-13 is 439 2021-12-02 07:18:56,323 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-806 is 325 2021-12-02 07:18:56,323 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-734 is 313 2021-12-02 07:18:56,323 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-446 is 342 2021-12-02 07:18:56,323 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-374 is 320 2021-12-02 07:18:56,325 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-230 is 325 2021-12-02 07:18:56,325 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-807 is 348 2021-12-02 07:18:56,325 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-735 is 332 2021-12-02 07:18:56,325 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-375 is 347 2021-12-02 07:18:56,325 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-231 is 346 2021-12-02 07:18:56,325 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-15 is 357 2021-12-02 07:18:56,325 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-808 is 381 2021-12-02 07:18:56,325 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-736 is 421 2021-12-02 07:18:56,325 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-448 is 356 2021-12-02 07:18:56,325 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-376 is 401 2021-12-02 07:18:56,325 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-232 is 385 2021-12-02 07:18:56,325 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-16 is 377 2021-12-02 07:18:56,326 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-737 is 310 2021-12-02 07:18:56,326 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-449 is 307 2021-12-02 07:18:56,326 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-377 is 340 2021-12-02 07:18:56,326 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-233 is 335 2021-12-02 07:18:56,326 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-17 is 357 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-810 is 358 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-738 is 418 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-450 is 312 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-378 is 413 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-234 is 317 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-18 is 310 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-811 is 345 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-739 is 338 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-451 is 341 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-379 is 384 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-235 is 386 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-19 is 301 2021-12-02 07:18:56,328 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-794 is 400 2021-12-02 07:18:56,329 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-722 is 383 2021-12-02 07:18:56,329 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-506 is 351 2021-12-02 07:18:56,329 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-434 is 380 2021-12-02 07:18:56,329 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-362 is 367 2021-12-02 07:18:56,329 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-218 is 372 2021-12-02 07:18:56,329 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-2 is 345 2021-12-02 07:18:56,331 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-812 is 338 2021-12-02 07:18:56,331 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-740 is 349 2021-12-02 07:18:56,331 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-452 is 332 2021-12-02 07:18:56,331 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-380 is 373 2021-12-02 07:18:56,331 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-236 is 364 2021-12-02 07:18:56,332 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-20 is 346 2021-12-02 07:18:56,332 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-813 is 369 2021-12-02 07:18:56,332 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-741 is 402 2021-12-02 07:18:56,332 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-453 is 361 2021-12-02 07:18:56,332 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-381 is 321 2021-12-02 07:18:56,332 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-237 is 395 2021-12-02 07:18:56,332 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-21 is 361 2021-12-02 07:18:56,332 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-814 is 357 2021-12-02 07:18:56,332 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-742 is 381 2021-12-02 07:18:56,332 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-454 is 382 2021-12-02 07:18:56,334 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-382 is 364 2021-12-02 07:18:56,334 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-238 is 388 2021-12-02 07:18:56,334 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-22 is 379 2021-12-02 07:18:56,334 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-815 is 332 2021-12-02 07:18:56,334 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-743 is 399 2021-12-02 07:18:56,334 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-455 is 366 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-383 is 387 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-239 is 375 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-23 is 381 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-816 is 402 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-744 is 382 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-456 is 364 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-384 is 362 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-240 is 363 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-24 is 338 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-745 is 395 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-385 is 385 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-241 is 332 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-25 is 362 2021-12-02 07:18:56,335 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-818 is 379 2021-12-02 07:18:56,336 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-746 is 373 2021-12-02 07:18:56,336 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-458 is 362 2021-12-02 07:18:56,336 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-386 is 413 2021-12-02 07:18:56,336 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-242 is 368 2021-12-02 07:18:56,336 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-26 is 368 2021-12-02 07:18:56,336 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-819 is 380 2021-12-02 07:18:56,336 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-747 is 357 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-387 is 410 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-243 is 443 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-171 is 421 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-27 is 326 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-820 is 394 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-748 is 379 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-460 is 372 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-388 is 375 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-244 is 340 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-172 is 350 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-28 is 404 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-821 is 407 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-749 is 391 2021-12-02 07:18:56,338 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-461 is 396 2021-12-02 07:18:56,339 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-389 is 345 2021-12-02 07:18:56,339 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-245 is 421 2021-12-02 07:18:56,339 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-173 is 374 2021-12-02 07:18:56,339 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-795 is 350 2021-12-02 07:18:56,339 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-723 is 356 2021-12-02 07:18:56,339 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-507 is 424 2021-12-02 07:18:56,339 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-435 is 392 2021-12-02 07:18:56,341 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-363 is 390 2021-12-02 07:18:56,341 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-219 is 375 2021-12-02 07:18:56,341 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-75 is 397 2021-12-02 07:18:56,341 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-3 is 333 2021-12-02 07:18:56,341 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-822 is 322 2021-12-02 07:18:56,341 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-750 is 358 2021-12-02 07:18:56,341 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-462 is 396 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-390 is 374 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-246 is 359 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-174 is 395 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-30 is 379 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-823 is 369 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-751 is 371 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-463 is 400 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-391 is 339 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-247 is 341 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-175 is 359 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-31 is 313 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-824 is 376 2021-12-02 07:18:56,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-752 is 364 2021-12-02 07:18:56,344 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-464 is 392 2021-12-02 07:18:56,344 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-392 is 364 2021-12-02 07:18:56,344 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-248 is 356 2021-12-02 07:18:56,344 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-176 is 303 2021-12-02 07:18:56,344 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-32 is 373 2021-12-02 07:18:56,344 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-825 is 322 2021-12-02 07:18:56,344 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-753 is 335 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-465 is 420 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-393 is 421 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-249 is 313 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-177 is 329 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-33 is 370 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-826 is 357 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-754 is 354 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-466 is 408 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-394 is 375 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-250 is 426 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-178 is 339 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-34 is 369 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-827 is 391 2021-12-02 07:18:56,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-755 is 407 2021-12-02 07:18:56,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-683 is 339 2021-12-02 07:18:56,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-467 is 344 2021-12-02 07:18:56,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-395 is 399 2021-12-02 07:18:56,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-251 is 390 2021-12-02 07:18:56,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-179 is 395 2021-12-02 07:18:56,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-35 is 445 2021-12-02 07:18:56,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-828 is 337 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-756 is 370 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-684 is 404 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-468 is 393 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-252 is 384 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-180 is 383 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-36 is 407 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-829 is 380 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-757 is 420 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-685 is 324 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-469 is 442 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-397 is 415 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-253 is 364 2021-12-02 07:18:56,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-181 is 434 2021-12-02 07:18:56,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-37 is 411 2021-12-02 07:18:56,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-758 is 337 2021-12-02 07:18:56,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-686 is 418 2021-12-02 07:18:56,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-470 is 406 2021-12-02 07:18:56,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-398 is 424 2021-12-02 07:18:56,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-254 is 335 2021-12-02 07:18:56,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-182 is 336 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-38 is 361 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-831 is 386 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-759 is 359 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-687 is 365 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-471 is 386 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-399 is 371 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-255 is 444 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-183 is 307 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-39 is 378 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-796 is 384 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-724 is 416 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-508 is 363 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-436 is 407 2021-12-02 07:18:56,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-364 is 438 2021-12-02 07:18:56,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-220 is 393 2021-12-02 07:18:56,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-76 is 370 2021-12-02 07:18:56,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-4 is 432 2021-12-02 07:18:56,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-832 is 320 2021-12-02 07:18:56,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-760 is 407 2021-12-02 07:18:56,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-688 is 389 2021-12-02 07:18:56,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-472 is 325 2021-12-02 07:18:56,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-400 is 349 2021-12-02 07:18:56,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-184 is 378 2021-12-02 07:18:56,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-40 is 334 2021-12-02 07:18:56,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-833 is 331 2021-12-02 07:18:56,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-761 is 358 2021-12-02 07:18:56,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-689 is 333 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-473 is 350 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-401 is 419 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-185 is 422 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-41 is 387 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-834 is 370 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-762 is 325 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-690 is 331 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-474 is 382 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-402 is 366 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-186 is 423 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-42 is 334 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-835 is 359 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-763 is 419 2021-12-02 07:18:56,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-691 is 316 2021-12-02 07:18:56,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-475 is 382 2021-12-02 07:18:56,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-403 is 346 2021-12-02 07:18:56,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-187 is 374 2021-12-02 07:18:56,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-43 is 408 2021-12-02 07:18:56,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-836 is 445 2021-12-02 07:18:56,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-764 is 360 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-692 is 353 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-476 is 374 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-404 is 315 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-188 is 413 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-44 is 401 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-837 is 316 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-765 is 316 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-693 is 410 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-477 is 376 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-405 is 397 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-189 is 385 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-45 is 360 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-838 is 401 2021-12-02 07:18:56,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-766 is 368 2021-12-02 07:18:56,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-694 is 359 2021-12-02 07:18:56,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-478 is 369 2021-12-02 07:18:56,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-406 is 407 2021-12-02 07:18:56,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-190 is 380 2021-12-02 07:18:56,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-46 is 382 2021-12-02 07:18:56,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-839 is 306 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-767 is 361 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-695 is 361 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-479 is 402 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-407 is 390 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-47 is 346 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-840 is 361 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-768 is 302 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-696 is 403 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-480 is 388 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-408 is 370 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-192 is 345 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-48 is 432 2021-12-02 07:18:56,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-841 is 393 2021-12-02 07:18:56,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-769 is 423 2021-12-02 07:18:56,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-697 is 366 2021-12-02 07:18:56,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-409 is 404 2021-12-02 07:18:56,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-193 is 361 2021-12-02 07:18:56,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-49 is 358 2021-12-02 07:18:56,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-797 is 417 2021-12-02 07:18:56,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-725 is 419 2021-12-02 07:18:56,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-509 is 437 2021-12-02 07:18:56,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-437 is 369 2021-12-02 07:18:56,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-365 is 382 2021-12-02 07:18:56,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-221 is 352 2021-12-02 07:18:56,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-77 is 444 2021-12-02 07:18:56,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-5 is 364 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-842 is 383 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-770 is 424 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-698 is 372 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-482 is 368 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-410 is 362 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-194 is 406 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-50 is 381 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-843 is 358 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-771 is 374 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-699 is 343 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-483 is 401 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-411 is 415 2021-12-02 07:18:56,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-195 is 430 2021-12-02 07:18:56,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-51 is 365 2021-12-02 07:18:56,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-844 is 421 2021-12-02 07:18:56,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-772 is 366 2021-12-02 07:18:56,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-700 is 370 2021-12-02 07:18:56,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-484 is 394 2021-12-02 07:18:56,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-412 is 359 2021-12-02 07:18:56,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-196 is 338 2021-12-02 07:18:56,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-52 is 380 2021-12-02 07:18:56,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-845 is 374 2021-12-02 07:18:56,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-773 is 304 2021-12-02 07:18:56,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-701 is 332 2021-12-02 07:18:56,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-485 is 426 2021-12-02 07:18:56,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-413 is 396 2021-12-02 07:18:56,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-197 is 397 2021-12-02 07:18:56,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-53 is 345 2021-12-02 07:18:56,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-846 is 352 2021-12-02 07:18:56,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-774 is 360 2021-12-02 07:18:56,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-702 is 395 2021-12-02 07:18:56,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-486 is 403 2021-12-02 07:18:56,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-414 is 357 2021-12-02 07:18:56,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-342 is 355 2021-12-02 07:18:56,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-198 is 387 2021-12-02 07:18:56,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-54 is 379 2021-12-02 07:18:56,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-847 is 394 2021-12-02 07:18:56,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-775 is 347 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-703 is 338 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-487 is 332 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-415 is 419 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-343 is 369 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-199 is 391 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-55 is 333 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-848 is 389 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-776 is 351 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-704 is 355 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-488 is 399 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-416 is 414 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-344 is 390 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-200 is 385 2021-12-02 07:18:56,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-56 is 402 2021-12-02 07:18:56,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-849 is 397 2021-12-02 07:18:56,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-777 is 415 2021-12-02 07:18:56,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-705 is 369 2021-12-02 07:18:56,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-489 is 337 2021-12-02 07:18:56,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-417 is 407 2021-12-02 07:18:56,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-345 is 393 2021-12-02 07:18:56,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-201 is 343 2021-12-02 07:18:56,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-57 is 428 2021-12-02 07:18:56,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-850 is 389 2021-12-02 07:18:56,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-778 is 406 2021-12-02 07:18:56,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-706 is 367 2021-12-02 07:18:56,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-490 is 404 2021-12-02 07:18:56,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-418 is 362 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-346 is 336 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-202 is 362 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-58 is 372 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-851 is 421 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-779 is 347 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-707 is 412 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-491 is 415 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-419 is 318 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-347 is 354 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-203 is 382 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-59 is 359 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-798 is 381 2021-12-02 07:18:56,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-726 is 368 2021-12-02 07:18:56,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-510 is 365 2021-12-02 07:18:56,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-438 is 379 2021-12-02 07:18:56,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-366 is 301 2021-12-02 07:18:56,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-222 is 336 2021-12-02 07:18:56,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-78 is 420 2021-12-02 07:18:56,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-852 is 340 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-780 is 373 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-708 is 318 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-492 is 325 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-420 is 407 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-204 is 389 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-60 is 329 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-781 is 392 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-709 is 322 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-493 is 373 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-421 is 445 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-349 is 343 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-205 is 401 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-61 is 379 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-782 is 405 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-710 is 397 2021-12-02 07:18:56,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-494 is 327 2021-12-02 07:18:56,380 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-422 is 308 2021-12-02 07:18:56,380 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-350 is 333 2021-12-02 07:18:56,380 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-206 is 301 2021-12-02 07:18:56,380 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-62 is 377 2021-12-02 07:18:56,380 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-783 is 342 2021-12-02 07:18:56,380 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-711 is 398 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-495 is 385 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-423 is 383 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-351 is 351 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-207 is 432 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-63 is 429 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-784 is 361 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-712 is 407 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-496 is 337 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-424 is 440 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-352 is 317 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-208 is 319 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-64 is 427 2021-12-02 07:18:56,381 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-785 is 343 2021-12-02 07:18:56,382 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-713 is 346 2021-12-02 07:18:56,382 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-497 is 334 2021-12-02 07:18:56,382 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-425 is 354 2021-12-02 07:18:56,382 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-353 is 359 2021-12-02 07:18:56,382 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-209 is 316 2021-12-02 07:18:56,382 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-65 is 343 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-786 is 397 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-714 is 339 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-498 is 413 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-426 is 396 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-354 is 384 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-210 is 402 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-66 is 377 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-787 is 345 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-715 is 354 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-499 is 451 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-427 is 326 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-355 is 335 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-211 is 457 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-67 is 401 2021-12-02 07:18:56,384 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-788 is 345 2021-12-02 07:18:56,385 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-716 is 337 2021-12-02 07:18:56,385 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-500 is 359 2021-12-02 07:18:56,385 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-428 is 452 2021-12-02 07:18:56,385 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-356 is 368 2021-12-02 07:18:56,385 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-212 is 347 2021-12-02 07:18:56,385 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-68 is 375 2021-12-02 07:18:56,385 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-789 is 382 2021-12-02 07:18:56,385 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-717 is 390 2021-12-02 07:18:56,385 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-501 is 363 2021-12-02 07:18:56,387 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-429 is 403 2021-12-02 07:18:56,387 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-357 is 349 2021-12-02 07:18:56,387 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-213 is 320 2021-12-02 07:18:56,387 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-69 is 377 2021-12-02 07:18:56,387 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-799 is 345 2021-12-02 07:18:56,387 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-727 is 379 2021-12-02 07:18:56,387 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-511 is 409 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-439 is 386 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-367 is 329 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-223 is 341 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-79 is 362 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-7 is 358 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-790 is 412 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-718 is 424 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-502 is 445 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-430 is 392 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-358 is 371 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-214 is 398 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-70 is 334 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-791 is 386 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-719 is 347 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-503 is 397 2021-12-02 07:18:56,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-431 is 429 2021-12-02 07:18:56,390 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-359 is 391 2021-12-02 07:18:56,390 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-215 is 407 2021-12-02 07:18:56,390 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-71 is 335 2021-12-02 07:18:56,390 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-800 is 355 2021-12-02 07:18:56,390 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-728 is 400 2021-12-02 07:18:56,390 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-512 is 391 2021-12-02 07:18:56,390 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-440 is 382 2021-12-02 07:18:56,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-368 is 401 2021-12-02 07:18:56,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-224 is 404 2021-12-02 07:18:56,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-80 is 413 2021-12-02 07:18:56,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-8 is 340 2021-12-02 07:18:56,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-801 is 339 2021-12-02 07:18:56,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-729 is 355 2021-12-02 07:18:56,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-441 is 357 2021-12-02 07:18:56,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-369 is 367 2021-12-02 07:18:56,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-225 is 370 2021-12-02 07:18:56,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-81 is 396 2021-12-02 07:18:56,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-9 is 389 2021-12-02 07:18:56,391 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:18:56,394 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:18:56,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:18:56,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:18:56,835 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:18:56,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:18:56,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:18:56,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:18:56,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:18:56,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:18:56,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:18:56,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:18:56,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-02 07:18:57,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-02 07:18:57,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-02 07:18:57,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-02 07:18:57,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-02 07:19:00,341 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.0/kvstore-648 is 386 2021-12-02 07:19:00,341 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.0/kvstore-576 is 303 2021-12-02 07:19:00,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.0/kvstore-216 is 346 2021-12-02 07:19:00,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.1/kvstore-721 is 301 2021-12-02 07:19:00,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.1/kvstore-217 is 333 2021-12-02 07:19:00,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-874 is 311 2021-12-02 07:19:00,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.11/kvstore-587 is 308 2021-12-02 07:19:00,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-660 is 313 2021-12-02 07:19:00,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-878 is 301 2021-12-02 07:19:00,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-302 is 301 2021-12-02 07:19:00,342 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.15/kvstore-663 is 312 2021-12-02 07:19:00,344 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-592 is 325 2021-12-02 07:19:00,344 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-304 is 300 2021-12-02 07:19:00,344 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.2/kvstore-722 is 313 2021-12-02 07:19:00,344 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.2/kvstore-650 is 318 2021-12-02 07:19:00,344 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-884 is 317 2021-12-02 07:19:00,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.21/kvstore-237 is 302 2021-12-02 07:19:00,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.22/kvstore-238 is 389 2021-12-02 07:19:00,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.23/kvstore-743 is 319 2021-12-02 07:19:00,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.23/kvstore-311 is 345 2021-12-02 07:19:00,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.23/kvstore-239 is 321 2021-12-02 07:19:00,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.25/kvstore-745 is 312 2021-12-02 07:19:00,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.26/kvstore-242 is 349 2021-12-02 07:19:00,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.27/kvstore-243 is 348 2021-12-02 07:19:00,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.28/kvstore-244 is 373 2021-12-02 07:19:00,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.28/kvstore-172 is 332 2021-12-02 07:19:00,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-677 is 301 2021-12-02 07:19:00,345 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-245 is 312 2021-12-02 07:19:00,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-173 is 340 2021-12-02 07:19:00,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-246 is 311 2021-12-02 07:19:00,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.31/kvstore-607 is 301 2021-12-02 07:19:00,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.31/kvstore-247 is 358 2021-12-02 07:19:00,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.32/kvstore-248 is 306 2021-12-02 07:19:00,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.32/kvstore-176 is 364 2021-12-02 07:19:00,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-537 is 362 2021-12-02 07:19:00,346 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-249 is 342 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-177 is 399 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-898 is 325 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-754 is 339 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-250 is 323 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-251 is 351 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-179 is 369 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-252 is 306 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-180 is 321 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.37/kvstore-541 is 319 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.37/kvstore-253 is 306 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-542 is 301 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-254 is 320 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-182 is 347 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-183 is 344 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-760 is 310 2021-12-02 07:19:00,348 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-328 is 307 2021-12-02 07:19:00,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-329 is 300 2021-12-02 07:19:00,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-257 is 352 2021-12-02 07:19:00,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.42/kvstore-762 is 300 2021-12-02 07:19:00,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.44/kvstore-908 is 333 2021-12-02 07:19:00,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.44/kvstore-188 is 333 2021-12-02 07:19:00,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-693 is 302 2021-12-02 07:19:00,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-333 is 302 2021-12-02 07:19:00,349 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-189 is 309 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.46/kvstore-622 is 305 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.46/kvstore-190 is 346 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-551 is 386 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-335 is 328 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-191 is 304 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-336 is 331 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-192 is 345 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-697 is 322 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-625 is 313 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-553 is 339 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-193 is 332 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.5/kvstore-221 is 314 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.51/kvstore-195 is 322 2021-12-02 07:19:00,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-700 is 347 2021-12-02 07:19:00,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-556 is 326 2021-12-02 07:19:00,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-340 is 308 2021-12-02 07:19:00,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-196 is 346 2021-12-02 07:19:00,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.53/kvstore-701 is 373 2021-12-02 07:19:00,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.53/kvstore-629 is 400 2021-12-02 07:19:00,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.54/kvstore-918 is 304 2021-12-02 07:19:00,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.54/kvstore-558 is 319 2021-12-02 07:19:00,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.54/kvstore-198 is 373 2021-12-02 07:19:00,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.55/kvstore-559 is 320 2021-12-02 07:19:00,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.55/kvstore-199 is 318 2021-12-02 07:19:00,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.56/kvstore-632 is 324 2021-12-02 07:19:00,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.56/kvstore-200 is 315 2021-12-02 07:19:00,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.57/kvstore-201 is 305 2021-12-02 07:19:00,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.60/kvstore-204 is 309 2021-12-02 07:19:00,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.61/kvstore-637 is 333 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-854 is 385 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-206 is 309 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.63/kvstore-711 is 309 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.63/kvstore-207 is 345 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.64/kvstore-208 is 351 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-929 is 316 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-281 is 301 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.66/kvstore-714 is 311 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.66/kvstore-642 is 309 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-859 is 303 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-571 is 335 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.68/kvstore-932 is 353 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-717 is 322 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-213 is 330 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.7/kvstore-583 is 332 2021-12-02 07:19:00,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.7/kvstore-223 is 332 2021-12-02 07:19:00,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-935 is 312 2021-12-02 07:19:00,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-647 is 315 2021-12-02 07:19:00,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-575 is 301 2021-12-02 07:19:00,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-215 is 433 2021-12-02 07:19:00,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-872 is 327 2021-12-02 07:19:00,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-656 is 301 2021-12-02 07:19:00,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-224 is 357 2021-12-02 07:19:00,357 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:19:00,358 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 12040877MB 2021-12-02 07:19:00,358 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 12293MB 2021-12-02 07:19:00,358 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 384967MB 2021-12-02 07:19:00,358 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 11655488MB 2021-12-02 07:19:00,361 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:19:00,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:19:00,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:19:00,786 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:19:00,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:19:00,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:19:00,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:19:00,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:19:00,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:19:00,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:19:00,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:19:00,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-02 07:19:00,924 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-02 07:19:00,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-02 07:19:01,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-02 07:19:01,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-02 07:19:02,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-02 07:19:02,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-02 07:19:02,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-02 07:19:03,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-02 07:19:03,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-02 07:19:03,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-02 07:19:04,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-02 07:19:04,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-02 07:19:04,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-02 07:19:05,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-02 07:19:05,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-02 07:19:06,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-02 07:19:06,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-02 07:19:06,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-02 07:19:07,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-02 07:19:07,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-02 07:19:08,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-02 07:19:08,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-02 07:19:08,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-02 07:19:09,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-02 07:19:09,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-02 07:19:09,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-02 07:19:10,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-02 07:19:10,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-02 07:19:10,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-02 07:19:11,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-02 07:19:11,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-02 07:19:12,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-02 07:19:12,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-02 07:19:12,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-02 07:19:13,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-02 07:19:13,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-02 07:19:13,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-02 07:19:14,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-02 07:19:14,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-02 07:19:15,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-02 07:19:15,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-02 07:19:15,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-02 07:19:16,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-02 07:19:16,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-02 07:19:16,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-02 07:19:17,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-02 07:19:17,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-02 07:19:18,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-02 07:19:18,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-02 07:19:18,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-02 07:19:19,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-02 07:19:19,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-02 07:19:19,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-02 07:19:20,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-02 07:19:20,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-02 07:19:21,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-02 07:19:21,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-02 07:19:21,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-02 07:19:22,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-02 07:19:22,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-02 07:19:22,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-02 07:19:23,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-02 07:19:23,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-02 07:19:24,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-02 07:19:24,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-02 07:19:24,765 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:19:24,769 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:19:24,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:19:25,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:19:25,102 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:19:25,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:19:25,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:19:25,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:19:25,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:19:25,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:19:25,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:19:25,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:19:25,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-02 07:19:25,230 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-02 07:19:25,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-02 07:19:25,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-02 07:19:26,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-02 07:19:26,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-02 07:19:26,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-02 07:19:27,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-02 07:19:27,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-02 07:19:27,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-02 07:19:28,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-02 07:19:28,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-02 07:19:29,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-02 07:19:29,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-02 07:19:29,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-02 07:19:30,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-02 07:19:30,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-02 07:19:31,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-02 07:19:31,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-02 07:19:31,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-02 07:19:32,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-02 07:19:32,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-02 07:19:32,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-02 07:19:33,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-02 07:19:33,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-02 07:19:34,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-02 07:19:34,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-02 07:19:34,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-02 07:19:35,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-02 07:19:35,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-02 07:19:35,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-02 07:19:36,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-02 07:19:36,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-02 07:19:37,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-02 07:19:37,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-02 07:19:37,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-02 07:19:38,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-02 07:19:38,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-02 07:19:38,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-02 07:19:39,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-02 07:19:39,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-02 07:19:39,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-02 07:19:40,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-02 07:19:40,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-02 07:19:40,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-02 07:19:41,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-02 07:19:41,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-02 07:19:42,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-02 07:19:42,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-02 07:19:42,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-02 07:19:43,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-02 07:19:43,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-02 07:19:44,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-02 07:19:44,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-02 07:19:44,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-02 07:19:45,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-02 07:19:45,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-02 07:19:45,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-02 07:19:46,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-02 07:19:46,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-02 07:19:46,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-02 07:19:47,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-02 07:19:47,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-02 07:19:47,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-02 07:19:48,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-02 07:19:48,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-02 07:19:49,117 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:19:49,118 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:19:49,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:19:49,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:19:49,473 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:19:49,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:19:49,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:19:49,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:19:49,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:19:49,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:19:49,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:19:49,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:19:49,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-02 07:19:49,657 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.68 - core(s): 72 2021-12-02 07:19:49,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-02 07:19:49,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-02 07:19:50,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-02 07:19:50,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-02 07:19:51,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-02 07:19:51,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-02 07:19:51,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-02 07:19:52,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-02 07:19:52,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-02 07:19:52,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-02 07:19:53,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-02 07:19:53,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-02 07:19:53,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-02 07:19:54,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-02 07:19:54,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-02 07:19:54,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-02 07:19:55,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-02 07:19:55,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-02 07:19:56,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-02 07:19:56,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-02 07:19:56,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-02 07:19:57,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-02 07:19:57,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-02 07:19:57,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-02 07:19:58,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-02 07:19:58,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-02 07:19:58,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-02 07:19:59,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-02 07:19:59,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-02 07:19:59,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-02 07:20:00,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-02 07:20:00,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-02 07:20:01,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-02 07:20:01,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-02 07:20:01,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-02 07:20:02,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-02 07:20:02,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-02 07:20:02,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-02 07:20:03,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-02 07:20:03,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-02 07:20:03,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-02 07:20:04,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-02 07:20:04,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-02 07:20:05,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-02 07:20:05,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-02 07:20:05,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-02 07:20:06,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-02 07:20:06,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-02 07:20:06,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-02 07:20:07,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-02 07:20:07,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-02 07:20:07,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-02 07:20:08,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-02 07:20:08,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-02 07:20:08,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-02 07:20:09,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-02 07:20:09,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-02 07:20:09,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-02 07:20:10,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-02 07:20:10,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-02 07:20:10,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-02 07:20:11,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-02 07:20:11,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-02 07:20:12,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-02 07:20:12,391 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:20:12,394 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:20:12,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:20:12,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:20:12,809 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:20:12,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:20:12,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:20:12,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:20:12,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:20:12,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:20:12,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:20:12,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:20:12,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-02 07:20:12,963 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-02 07:20:12,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-02 07:20:13,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-02 07:20:13,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-02 07:20:14,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-02 07:20:14,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-02 07:20:14,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-02 07:20:15,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-02 07:20:15,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-02 07:20:15,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-02 07:20:16,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-02 07:20:16,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-02 07:20:16,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-02 07:20:17,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-02 07:20:17,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-02 07:20:18,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-02 07:20:18,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-02 07:20:18,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-02 07:20:19,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-02 07:20:19,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-02 07:20:20,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-02 07:20:20,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-02 07:20:20,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-02 07:20:21,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-02 07:20:21,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-02 07:20:21,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-02 07:20:22,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-02 07:20:22,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-02 07:20:23,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-02 07:20:23,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-02 07:20:23,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-02 07:20:24,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-02 07:20:24,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-02 07:20:24,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-02 07:20:25,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-02 07:20:25,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-02 07:20:25,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-02 07:20:26,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-02 07:20:26,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-02 07:20:27,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-02 07:20:27,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-02 07:20:27,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-02 07:20:28,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-02 07:20:28,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-02 07:20:29,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-02 07:20:29,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-02 07:20:29,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-02 07:20:30,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-02 07:20:30,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-02 07:20:30,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-02 07:20:31,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-02 07:20:31,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-02 07:20:32,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-02 07:20:32,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-02 07:20:32,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-02 07:20:33,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-02 07:20:33,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-02 07:20:33,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-02 07:20:34,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-02 07:20:34,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-02 07:20:34,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-02 07:20:35,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-02 07:20:35,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-02 07:20:36,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-02 07:20:36,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-02 07:20:36,750 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:20:36,750 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 647089.1875, u'ReplayWALOffset': u'9168:17338368', u'NTablesCreated': 607135, u'RecentReadIOAmpSet': 0.19857272505760193, u'ReadIOAmp': 2784.32958984375, u'BlockCacheMisses': 58617581, u'NFlushes': 143867, u'RecentWriteAmp': 2.4100801944732666, u'NumSyncFlushes': 2085, u'TotalMemUsed': 129014262, u'NBloomFilterHits': 674732781, u'Fragmentation': 0.4473702609539032, u'NumNormalFlushes': 78314, u'RecentBlockCacheHitRatio': 90.99027252197266, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 326402, u'ReadIOAmp': 1.0087797035392108, u'MemtablesMemUsed': 2071104, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 357150803, u'CheckpointOverhead': 10747904, u'NItemsGet': 874517, u'NWriteIOs': 490644, u'NReadBytes': 30487957133L, u'LevelStats': [{u'TargetSize': 33554432, u'LogicalSize': 15823222, u'SizeRatio': 0.471568763256073, u'NWriteBytes': 683749376, u'BloomFilterMemUsed': 0, u'NumTables': 57, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 704512}, {u'TargetSize': 33554432, u'LogicalSize': 22685206, u'SizeRatio': 0.67607182264328, u'NWriteBytes': 571940864, u'BloomFilterMemUsed': 0, u'NumTables': 15, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 577536}, {u'TargetSize': 506461666, u'LogicalSize': 491115480, u'SizeRatio': 0.9696992151030834, u'NWriteBytes': 1988116480, u'BloomFilterMemUsed': 0, u'NumTables': 173, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 13275136}, {u'TargetSize': 25798224090L, u'LogicalSize': 23529797572L, u'SizeRatio': 0.9120704390315264, u'NWriteBytes': 2907607040L, u'BloomFilterMemUsed': 0, u'NumTables': 895, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 234024960}, {u'TargetSize': 101292333818L, u'LogicalSize': 51596448181L, u'SizeRatio': 0.5093815714988604, u'NWriteBytes': 45029269504L, u'BloomFilterMemUsed': 0, u'NumTables': 190, u'WastedSpace': 950281272, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 51596447744L}], u'NItemsGarbageCollected': 17596691, u'NInternalFragmentationCompacts': 35, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 316290109, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 154560, u'NNonL0LocalCompacts': 19, u'NWriterCompacts': 0, u'NReadIOs': 10249998, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 454020, u'CompactWriter': {u'NDataBlockWrites': 3514077, u'NWriteBytes': 10451025920L, u'NIndexBlockWrites': 167209, u'NBytesIncoming': 14394718537L, u'NSyncs': 0, u'NInserts': 522264607, u'NWriteIOs': 158358}, u'NWriteBytes': 81341231529L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 882195, u'NReadBytes': 2819506461L, u'NDataBlockReads': 874517, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2623551}, u'NBloomFilterMisses': 0, u'NItemsMerged': 16404770, u'LogicalDiskSize': 51596448181L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.8630894569149922, u'BufferMemUsed': 352334, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 28066650609L, u'NTableFiles': 1572, u'NFileCountCompacts': 336, u'TableObjectMemUsed': 399280, u'TreeSnapshotMemoryUsed': 161744, u'TotalDiskUsage': 58834563072L, u'BytesIncoming': 94244265038L, u'NItemsDel': 0, u'PendingCheckpointBytes': 39899136, u'NCompacts': 45579, u'NItemsPut': 99754969, u'NSyncs': 65559, u'CompactReader': {u'NReadIOs': 9367499, u'NReadBytes': 27668267274L, u'NDataBlockReads': 9114483, u'NBytesOutgoing': 0, u'NIndexBlockReads': 253016}, u'ActiveDiskUsage': 51845029888L, u'NTablesDeleted': 199781, u'IterateReader': {u'NReadIOs': 304, u'NReadBytes': 183398, u'NDataBlockReads': 152, u'NBytesOutgoing': 0, u'NIndexBlockReads': 152}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 21465631, u'NWriteBytes': 70890205609L, u'NIndexBlockWrites': 538974, u'NBytesIncoming': 93399036786L, u'NSyncs': 65559, u'NInserts': 98229771, u'NWriteIOs': 332286}}, u'NReadBytes': 67810907233639L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 108.63333129882812, u'NInternalFragmentationCompacts': 35, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 335628772, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 5120, u'NTxns': 9557203, u'NWriteBytes': 156705443840L, u'NSegments': 9170, u'NSyncs': 9557203, u'NLoggedBytes': 156705443840L, u'NActiveSegments': 3, u'NWrites': 9581301, u'NInserts': 166689604, u'DiskUsed': 44040192}, u'NGets': 9118406, u'NReadBytesSet': 62656165386L, u'NReadBytesCompact': 75313892476L, u'IndexResidentRatio': 0.06605975329875946, u'CompactQueueSize': 0, u'NSyncsPerSec': 0.6000000238418579, u'BlockCacheHitRatio': 73.70966339111328, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 67735593006499L, u'BloomFilterMemUsedRatio': 0.16169048845767975, u'NWriteBytes': 290384453346L, u'BytesOutgoing': 1232842818, u'RecentReadIOAmp': 2.175769090652466, u'WriteAmp': 2.9867665767669678, u'NBlocksCached': 34301, u'LogicalDataSize': 29211138910L, u'ActiveBloomFilterMemUsed': 33884552, u'NFileCountCompacts': 15145, u'NTableFiles': 2315, u'FlushQueueSize': 72, u'TableObjectMemUsed': 583544, u'BlockCacheQuota': 99116234, u'TotalDiskUsage': 60365832192L, u'NPurgerRuns': 212, u'ReadIOAmpSet': 0.0, u'NSyncs': 196677, u'NReadIOsSetPerSec': 58.43333435058594, u'NReadIOsPerSec': 233.46665954589844, u'NTablesDeleted': 480142, u'BloomFilterFPR': 0.01531189400702715, u'NBloomFilterHitsPerSec': 2288.433349609375, u'WriteCacheMemUsed': 3133120, u'RecentBloomFilterFPR': 0.01662990264594555, u'BytesPerRead': 7421575.0, u'BytesIncomingPerSec': 399559.46875, u'NReadIOsGetPerSec': 231.06666564941406, u'NGetStats': 482465, u'TotalIndexBlocksSize': 378053006, u'NGetStatsPerSec': 0.8333333134651184, u'NWriteBytesCompact': 56501350400L, u'NReadIOsSet': 23526362, u'ActiveDataSize': 29211138910L, u'NWriteBytesCompactPerSec': 546.1333618164062, u'NBloomFilterMissesPerSec': 38.70000076293945, u'NCommitBatches': 7314245, u'NGetStatsComputed': 482465, u'NWriteIOs': 10564634, u'NReadIOsGet': 25412174491L, u'RecentReadAmpGet': 6.316378593444824, u'WriteCacheQuota': 24779060, u'LSMTreeObjectMemUsed': 132608, u'NSetsPerSec': 294.26666259765625, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.46666666865348816, u'ManagedFileObjectMemUsed': 225888, u'ReadAmpGet': 54891.77734375, u'BlockCacheMemUsed': 99191860, u'NWriterCompacts': 0, u'NReadIOs': 25439524981L, u'WALBufferMemUsed': 65536, u'ReadAmp': 54952.8671875, u'keyStats': {u'NItemsTransformed': 1099893401, u'NTablesCreated': 147510, u'ReadIOAmp': 337.82726890199444, u'MemtablesMemUsed': 219712, u'NBloomFilterHits': 674731312, u'TotalIndexBlocksSize': 20891427, u'CheckpointOverhead': 82190336, u'NItemsGet': 75219721, u'NWriteIOs': 293907, u'NReadBytes': 67780261475029L, u'LevelStats': [{u'TargetSize': 33554432, u'LogicalSize': 1546769, u'SizeRatio': 0.04609730839729309, u'NWriteBytes': 3229417472L, u'BloomFilterMemUsed': 21624, u'NumTables': 49, u'WastedSpace': 126, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1482752}, {u'TargetSize': 33554432, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 33554432, u'LogicalSize': 25320543, u'SizeRatio': 0.7546109855175018, u'NWriteBytes': 12518535168L, u'BloomFilterMemUsed': 934848, u'NumTables': 16, u'WastedSpace': 55701, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 25370624}, {u'TargetSize': 126188690, u'LogicalSize': 117490276, u'SizeRatio': 0.9310681963653002, u'NWriteBytes': 6370467840L, u'BloomFilterMemUsed': 5009408, u'NumTables': 61, u'WastedSpace': 305010, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 117788672}, {u'TargetSize': 1261886935, u'LogicalSize': 1261886935, u'SizeRatio': 1.0, u'NWriteBytes': 10140954624L, u'BloomFilterMemUsed': 27917312, u'NumTables': 169, u'WastedSpace': 4431107, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1266315264}], u'NItemsGarbageCollected': 31060447, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 19334967, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 50208, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 25428989288L, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 883824, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 188808, u'CompactWriter': {u'NDataBlockWrites': 16822389, u'NWriteBytes': 45989720064L, u'NIndexBlockWrites': 217019, u'NBytesIncoming': 64103991962L, u'NSyncs': 0, u'NInserts': 1109175410, u'NWriteIOs': 102284}, u'NWriteBytes': 51212732237L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 25411272913L, u'NReadBytes': 67732745964803L, u'NDataBlockReads': 48205497666L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 94589407471L}, u'NBloomFilterMisses': 10491973, u'NItemsMerged': 0, u'LogicalDiskSize': 1261886935, u'TotalBloomFilterMemUsed': 24111672, u'WriteAmp': 15.78682899323994, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 33883192, u'LogicalDataSize': 1144396659, u'NTableFiles': 523, u'NFileCountCompacts': 7411, u'TableObjectMemUsed': 129704, u'TreeSnapshotMemoryUsed': 67488, u'TotalDiskUsage': 1529466880, u'BytesIncoming': 3244016405L, u'NItemsDel': 0, u'PendingCheckpointBytes': 71589888, u'NCompacts': 9827, u'NItemsPut': 66101069, u'NSyncs': 65559, u'CompactReader': {u'NReadIOs': 17716297, u'NReadBytes': 47515358960L, u'NDataBlockReads': 17379155, u'NBytesOutgoing': 0, u'NIndexBlockReads': 337142}, u'ActiveDiskUsage': 1410957312, u'NTablesDeleted': 147205, u'IterateReader': {u'NReadIOs': 78, u'NReadBytes': 151266, u'NDataBlockReads': 39, u'NBytesOutgoing': 0, u'NIndexBlockReads': 39}, u'BloomFilterFPR': 0.0153117578308799, u'Writer': {u'NDataBlockWrites': 1081676, u'NWriteBytes': 5223012173L, u'NIndexBlockWrites': 126389, u'NBytesIncoming': 3769688240L, u'NSyncs': 65559, u'NInserts': 64907354, u'NWriteIOs': 191623}}, u'NReadBytesCompactPerSec': 1149.63330078125, u'TableMetaMemUsed': 705964, u'NDeletes': 0, u'BlockCacheHitsPerSec': 1097.0999755859375, u'BytesOutgoingPerSec': 102446.234375, u'NBloomFilterMisses': 10492091, u'LogicalDiskSize': 52858426758L, u'NSets': 66101069, u'TotalBloomFilterMemUsed': 24113032, u'BufferMemUsed': 352334, u'RecentReadAmp': 6.327600002288818, u'NGetsPerSec': 106.19999694824219, u'NInserts': 0, u'NCommitBatchesPerSec': 191.6999969482422, u'FragmentationWithOverhead': 0.516098141670227, u'NCompactsPerSec': 0.06666667014360428, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 648238.8125, u'TreeSnapshotMemoryUsed': 261056, u'BytesIncoming': 97223680098L, u'NCompacts': 62804, u'NumPersistentFlushes': 63468, u'NWriteIOsPerSec': 194.1666717529297, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 6093.11865234375, u'ActiveDiskUsage': 53256601600L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.8333333134651184, u'NReadBytesSetPerSec': 153158.265625, u'BlockCacheHits': 164344873, u'NWriteBytesPerSec': 962970.3125, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 133223, u'ReadIOAmp': 0.0013231755393768112, u'MemtablesMemUsed': 842304, u'NBloomFilterHits': 1469, u'TotalIndexBlocksSize': 10776, u'CheckpointOverhead': 1187840, u'NItemsGet': 14648850, u'NWriteIOs': 198782, u'NReadBytes': 157801477, u'LevelStats': [{u'TargetSize': 33554432, u'LogicalSize': 622051, u'SizeRatio': 0.018538564443588257, u'NWriteBytes': 649859072, u'BloomFilterMemUsed': 848, u'NumTables': 67, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 548864}, {u'TargetSize': 33554432, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 33554432, u'LogicalSize': 91642, u'SizeRatio': 0.002731144428253174, u'NWriteBytes': 38207488, u'BloomFilterMemUsed': 512, u'NumTables': 8, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 65536}], u'NItemsGarbageCollected': 35683066, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 3696, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 21120, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 285695, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 63136, u'CompactWriter': {u'NDataBlockWrites': 7398, u'NWriteBytes': 60604416, u'NIndexBlockWrites': 7398, u'NBytesIncoming': 33572780, u'NSyncs': 0, u'NInserts': 392094, u'NWriteIOs': 7398}, u'NWriteBytes': 1125045740, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 19383, u'NReadBytes': 27535235, u'NDataBlockReads': 7306642, u'NBytesOutgoing': 0, u'NIndexBlockReads': 7306642}, u'NBloomFilterMisses': 118, u'NItemsMerged': 0, u'LogicalDiskSize': 91642, u'TotalBloomFilterMemUsed': 1360, u'WriteAmp': 0.2672236052852835, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1360, u'LogicalDataSize': 91642, u'NTableFiles': 220, u'NFileCountCompacts': 7398, u'TableObjectMemUsed': 54560, u'TreeSnapshotMemoryUsed': 31824, u'TotalDiskUsage': 1802240, u'BytesIncoming': 4210128588L, u'NItemsDel': 0, u'PendingCheckpointBytes': 8192, u'NCompacts': 7398, u'NItemsPut': 35686417, u'NSyncs': 65559, u'CompactReader': {u'NReadIOs': 266312, u'NReadBytes': 130266242, u'NDataBlockReads': 133156, u'NBytesOutgoing': 0, u'NIndexBlockReads': 133156}, u'ActiveDiskUsage': 614400, u'NTablesDeleted': 133156, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.074354127284184, u'Writer': {u'NDataBlockWrites': 125825, u'NWriteBytes': 1064441324, u'NIndexBlockWrites': 125825, u'NBytesIncoming': 92164276, u'NSyncs': 65559, u'NInserts': 1209568, u'NWriteIOs': 191384}}}}} 2021-12-02 07:20:36,755 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.68': [0.02711552381515503, 0.028197159990668297, 0.027931896969676018, 0.0280121099203825, 0.02810099720954895, 0.028225567191839218, 0.029050393030047417, 0.029027940705418587, 0.02913789451122284, 0.026649577543139458, 0.026672279462218285, 0.027442270889878273, 0.027865629643201828, 0.028443701565265656, 0.026261696591973305, 0.026652906090021133, 0.023436494171619415, 0.023339537903666496, 0.024104701355099678, 0.024868236854672432, 0.0245573278516531, 0.026446163654327393, 0.024287113919854164, 0.02464996464550495, 0.024960575625300407, 0.025505734607577324, 0.025743085891008377, 0.022335005924105644, 0.02236221171915531, 0.022538136690855026, 0.023049505427479744, 0.023464549332857132, 0.02378845401108265, 0.024955756962299347, 0.025407278910279274, 0.021821683272719383, 0.022724779322743416, 0.023576799780130386, 0.02399004064500332, 0.024418609216809273, 0.021660685539245605, 0.02235471084713936, 0.023264143615961075, 0.023411352187395096, 0.023241795599460602, 0.02379503659904003, 0.024656206369400024, 0.02547939121723175, 0.023333508521318436, 0.02405140921473503, 0.024474719539284706, 0.025109607726335526, 0.025338713079690933, 0.02545810304582119, 0.023219184949994087, 0.023736419156193733, 0.02302919141948223, 0.023573247715830803, 0.024188319221138954, 0.023867828771471977, 0.02456517703831196, 0.024166038259863853, 0.025550317019224167, 0.026168204843997955], '172.23.110.65': [0.4473702609539032, 0.43465134501457214, 0.43647053837776184, 0.4175218343734741, 0.46953248977661133, 0.42909005284309387, 0.4311947822570801, 0.4247380793094635, 0.39935147762298584, 0.4343300759792328, 0.42207932472229004, 0.4426690340042114, 0.4204570949077606, 0.4558969736099243, 0.44602635502815247, 0.43338319659233093, 0.4374440908432007, 0.4611746370792389, 0.46690064668655396, 0.45975109934806824, 0.4652167856693268, 0.45863640308380127, 0.4576265811920166, 0.4783290922641754, 0.48227909207344055, 0.4454675614833832, 0.45613595843315125, 0.41644471883773804, 0.47678089141845703, 0.44807133078575134, 0.46190783381462097, 0.46047237515449524, 0.4578005075454712, 0.46130794286727905, 0.46438246965408325, 0.2891658842563629, 0.4730822741985321, 0.477197527885437, 0.4606493413448334, 0.4520007073879242, 0.46702152490615845, 0.4353877902030945, 0.46304455399513245, 0.44161301851272583, 0.46457231044769287, 0.48008841276168823, 0.4475966989994049, 0.4593370854854584, 0.46518275141716003, 0.47437992691993713, 0.46506187319755554, 0.45704808831214905, 0.4531066417694092, 0.4320218563079834, 0.445478618144989, 0.4446009397506714, 0.4513426125049591, 0.4482667148113251, 0.45235633850097656, 0.44853150844573975, 0.4384101629257202, 0.37664055824279785, 0.4406607151031494, 0.437797486782074], '172.23.110.66': [0.4286445379257202, 0.4674588143825531, 0.4228239059448242, 0.4328992962837219, 0.42030033469200134, 0.46607446670532227, 0.46521493792533875, 0.46124354004859924, 0.4370533525943756, 0.42880457639694214, 0.428431898355484, 0.467237651348114, 0.42559900879859924, 0.4208974540233612, 0.4292584955692291, 0.4514540731906891, 0.458941251039505, 0.4144723117351532, 0.45897313952445984, 0.4569227695465088, 0.4406488537788391, 0.4448419511318207, 0.40677711367607117, 0.47261106967926025, 0.44417011737823486, 0.37059926986694336, 0.4272502362728119, 0.44316384196281433, 0.4339946210384369, 0.44405508041381836, 0.4061790108680725, 0.4369488060474396, 0.38240647315979004, 0.42618075013160706, 0.41874611377716064, 0.4280535578727722, 0.40234851837158203, 0.41941627860069275, 0.44131359457969666, 0.4183579385280609, 0.42424941062927246, 0.4068453907966614, 0.40837255120277405, 0.41685548424720764, 0.4547557234764099, 0.4213292598724365, 0.4215105175971985, 0.44611984491348267, 0.44058868288993835, 0.4349082410335541, 0.4312775433063507, 0.4514743387699127, 0.4577501714229584, 0.44452422857284546, 0.42423349618911743, 0.45225784182548523, 0.4513069689273834, 0.4373430609703064, 0.4684086740016937, 0.4202115833759308, 0.43646177649497986, 0.42289769649505615, 0.43056705594062805, 0.46963900327682495], '172.23.110.64': [0.46595248579978943, 0.453651487827301, 0.460864394903183, 0.4551190733909607, 0.4638749659061432, 0.46408915519714355, 0.4336063265800476, 0.4577324688434601, 0.4537997841835022, 0.47108781337738037, 0.42629578709602356, 0.45412683486938477, 0.4659769833087921, 0.44206470251083374, 0.4283805191516876, 0.40717390179634094, 0.47260406613349915, 0.4355703890323639, 0.4509623646736145, 0.43294525146484375, 0.4734934866428375, 0.4510323107242584, 0.41972050070762634, 0.45857471227645874, 0.46098804473876953, 0.4477200508117676, 0.4829166829586029, 0.44867977499961853, 0.44170188903808594, 0.46506011486053467, 0.45677340030670166, 0.46833646297454834, 0.43247976899147034, 0.44422250986099243, 0.4530516266822815, 0.43932434916496277, 0.4351094663143158, 0.4667147696018219, 0.44134777784347534, 0.4246329367160797, 0.42948997020721436, 0.4158402979373932, 0.4346184730529785, 0.46612992882728577, 0.46817564964294434, 0.43135175108909607, 0.4436805844306946, 0.4389677941799164, 0.4267946779727936, 0.4485883116722107, 0.4738560914993286, 0.45545056462287903, 0.45140188932418823, 0.43234115839004517, 0.44382089376449585, 0.4647458493709564, 0.42427366971969604, 0.44823798537254333, 0.45510241389274597, 0.45077353715896606, 0.4324515163898468, 0.44383707642555237, 0.4404917061328888, 0.4440147578716278]} 2021-12-02 07:20:36,984 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 37.0 2021-12-02 07:20:37,196 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 37.0 2021-12-02 07:20:37,493 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.68 is 37.0 2021-12-02 07:20:37,766 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 37.0 2021-12-02 07:20:37,766 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.68': 37.0, '172.23.110.65': 37.0, '172.23.110.66': 37.0, '172.23.110.64': 37.0} ############################################################ # # Step 5: Crash Magma/memc with Loading of docs # ############################################################ ############################################################ # # RAM FootPrint: 6073.76171875 # ############################################################ 2021-12-02 07:20:37,822 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 07:20:37,822 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 86 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 2021-12-02 07:22:03,828 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 07:22:03,828 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 07:22:08,832 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:22:08,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:22:09,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:22:09,174 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:22:09,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:22:09,186 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:22:09,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:22:09,209 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:22:09,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:22:09,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:22:09,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:22:09,255 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:22:09,255 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:22:09,341 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['219151 memcached\n'], []) 2021-12-02 07:22:09,341 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 07:22:09,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:22:09,466 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['219151 memcached\n'], []) 2021-12-02 07:22:09,466 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:22:09,470 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:22:09,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:22:09,790 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:22:09,854 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:22:09,854 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:22:09,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:22:09,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:22:09,908 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:22:09,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:22:09,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:22:09,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:22:09,959 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:22:09,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:22:10,015 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['4442 memcached\n'], []) 2021-12-02 07:22:10,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 07:22:10,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:22:10,117 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['4442 memcached\n'], []) 2021-12-02 07:22:10,117 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:22:10,121 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:22:10,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:22:10,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:22:10,576 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:22:10,576 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:22:10,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:22:10,612 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:22:10,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:22:10,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:22:10,670 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:22:10,686 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:22:10,700 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:22:10,701 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:22:10,763 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['149378 memcached\n'], []) 2021-12-02 07:22:10,765 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 07:22:10,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:22:10,885 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['149378 memcached\n'], []) 2021-12-02 07:22:10,885 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:22:10,888 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:22:11,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:22:11,257 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:22:11,352 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:22:11,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:22:11,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:22:11,397 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:22:11,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:22:11,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:22:11,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:22:11,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:22:11,503 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:22:11,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:22:11,582 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['294941 memcached\n'], []) 2021-12-02 07:22:11,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 07:22:11,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:22:11,726 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['294941 memcached\n'], []) 2021-12-02 07:22:11,727 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:22:11,730 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:22:11,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:22:12,099 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:22:12,174 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:22:12,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:22:12,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:22:12,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:22:12,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:22:12,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:22:12,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:22:12,288 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:22:12,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:22:12,325 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 07:22:12,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:22:12,390 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:12,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:12,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:22:12,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:22:12,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:12,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:12,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:22:12,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:22:12,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:12,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:12,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:22:12,836 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:22:12,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:12,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:12,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:22:12,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:22:12,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:22:12,957 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 07:22:12,959 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:22:12,971 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:12,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:22:13,006 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 07:22:13,006 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:22:13,009 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:22:13,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:22:20,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:22:21,013 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:22:21,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:22:21,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:22:21,051 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:22:21,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:22:21,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:22:21,109 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:22:21,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:22:21,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:22:21,160 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 07:22:21,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:22:21,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:21,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:21,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:22:21,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:22:21,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:21,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:21,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:22:21,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:22:21,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:22:21,480 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 07:22:21,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:22:21,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:21,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:22:21,532 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 07:22:21,532 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:22:21,538 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:22:21,670 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:22:21,931 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:22:22,016 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:22:22,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:22:22,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:22:22,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:22:22,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:22:22,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:22:22,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:22:22,127 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:22:22,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:22:22,160 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 07:22:22,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:22:22,216 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:22,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:22,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:22:22,357 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:22:22,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:22,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:22,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:22:22,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:22:22,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:22,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:22,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:22:22,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:22:22,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:22,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:22,818 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:22:22,854 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:22:22,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:22,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:22,969 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:22:23,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:22:23,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:23,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:23,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:22:23,147 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:22:23,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:23,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:23,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:22:23,257 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:22:23,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:22:23,299 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 07:22:23,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:22:23,319 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:23,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:22:23,355 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 07:22:23,355 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:22:23,358 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:22:23,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:22:23,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:22:23,844 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:22:23,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:22:23,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:22:23,880 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:22:23,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:22:23,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:22:23,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:22:23,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:22:23,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:22:23,990 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 07:22:24,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:22:24,045 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:24,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:24,137 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:22:24,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:22:24,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:24,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:24,295 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:22:24,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:22:24,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:24,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:24,446 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:22:24,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:22:24,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:24,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:22:24,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:22:24,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:22:24,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:22:24,601 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 07:22:24,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:22:24,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:22:24,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:22:24,651 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 07:22:24,651 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:22:24,653 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:22:24,655 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:22:24,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:22:25,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:22:25,095 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:22:25,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:22:25,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:22:25,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:22:25,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:22:25,163 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:22:25,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:22:25,186 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:22:25,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:25,315 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:22:25,322 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:25,322 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:27,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:27,489 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:22:27,494 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:27,494 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:29,494 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:29,671 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:22:29,676 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:29,676 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:31,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:31,842 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:22:31,846 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:31,848 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:33,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:34,013 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:22:34,019 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:34,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:36,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:36,206 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:22:36,210 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:36,210 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:38,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:38,345 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:22:38,351 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:38,351 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:40,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:40,520 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:22:40,526 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:40,526 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:42,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:42,703 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:22:42,707 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:42,707 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:44,709 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:44,871 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:22:44,877 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:44,877 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:46,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:47,048 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:22:47,052 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:47,052 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:49,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:49,220 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:22:49,226 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:49,226 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:51,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:51,381 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:51,381 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:53,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:53,525 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:53,525 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:55,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:55,703 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:55,703 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:57,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:22:57,851 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:22:57,851 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:22:59,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:00,029 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:00,029 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:02,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:02,209 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:02,209 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:04,209 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:04,355 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:04,355 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:06,357 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:06,506 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:06,507 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:08,507 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:08,726 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:08,726 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:10,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:10,947 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:10,947 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:12,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:13,148 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:13,148 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:15,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:15,319 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:15,319 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:17,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:17,474 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:17,474 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:19,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:19,703 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:19,704 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:21,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:21,931 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:21,931 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:23,933 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:24,171 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:24,171 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:26,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:26,414 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:26,415 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:28,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:28,632 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:28,632 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:30,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:30,858 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:30,858 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:32,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:33,095 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:33,095 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:35,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:35,312 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:35,312 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:37,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:37,542 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:37,542 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:39,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:39,742 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:39,742 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:41,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:41,947 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:41,947 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:43,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:44,161 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:44,161 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:46,163 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:46,335 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:46,335 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:48,336 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:48,565 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:48,565 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:50,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:50,782 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:50,782 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:52,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:53,012 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:53,012 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:55,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:55,210 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:55,210 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:57,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:57,411 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:57,411 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:23:59,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:23:59,559 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:23:59,561 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:01,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:01,729 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:01,730 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:03,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:03,890 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:03,890 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:05,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:06,052 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:06,052 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:08,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:08,214 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:08,214 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:10,216 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:10,380 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:10,380 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:12,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:12,581 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:12,582 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:14,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:14,746 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:14,746 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:16,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:16,915 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:16,915 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:18,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:19,086 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:19,086 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:21,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:21,266 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:21,266 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:23,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:23,506 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:23,507 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:25,507 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:25,740 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:25,740 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:27,742 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:27,910 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:27,910 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:29,910 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:30,081 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:30,081 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:32,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:32,240 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:32,240 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:34,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:34,430 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:34,430 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:36,431 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:36,608 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:24:36,608 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:24:38,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:38,756 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:24:38,756 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 134.103999853. 2021-12-02 07:24:38,759 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:24:38,760 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:24:38,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:24:39,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:24:39,276 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:24:39,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:24:39,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:24:39,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:24:39,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:24:39,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:24:39,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:24:39,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:24:39,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:39,559 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:24:39,559 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.800999879837. 2021-12-02 07:24:39,561 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:24:39,562 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:24:39,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:24:39,992 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:24:40,072 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:24:40,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:24:40,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:24:40,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:24:40,127 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:24:40,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:24:40,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:24:40,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:24:40,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:40,321 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:24:40,322 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.760999917984. 2021-12-02 07:24:40,322 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:24:40,326 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:24:40,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:24:40,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:24:40,897 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:24:40,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:24:40,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:24:40,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:24:40,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:24:40,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:24:40,987 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:24:41,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:24:41,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:24:41,177 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:24:41,178 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.855999946594. ############################################################ # # RAM FootPrint: 6048.82421875 # ############################################################ 2021-12-02 07:24:41,213 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 07:24:41,213 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 99 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 2021-12-02 07:26:20,220 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 07:26:20,220 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 07:26:25,221 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:26:25,535 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:26:45,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:26:45,079 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:26:45,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:26:45,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:26:45,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:26:45,122 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:26:45,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:26:45,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:26:45,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:26:45,237 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:26:45,237 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:26:45,298 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['5131 memcached\n'], []) 2021-12-02 07:26:45,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 07:26:45,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:26:45,483 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['5131 memcached\n'], []) 2021-12-02 07:26:45,484 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:26:45,486 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:26:45,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:26:45,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:26:45,924 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:26:45,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:26:45,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:26:45,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:26:45,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:26:46,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:26:46,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:26:46,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:26:46,046 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:26:46,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:26:46,098 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['92684 memcached\n'], []) 2021-12-02 07:26:46,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 07:26:46,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:26:46,247 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['92684 memcached\n'], []) 2021-12-02 07:26:46,249 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:26:46,252 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:26:46,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:26:46,655 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:26:46,740 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:26:46,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:26:46,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:26:46,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:26:46,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:26:46,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:26:46,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:26:46,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:26:46,888 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:26:46,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:26:46,966 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['217279 memcached\n'], []) 2021-12-02 07:26:46,966 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 07:26:47,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:26:47,069 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['217279 memcached\n'], []) 2021-12-02 07:26:47,069 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:26:47,072 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:26:48,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:26:48,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:26:48,671 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:26:48,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:26:48,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:26:48,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:26:48,872 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:26:48,897 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:26:48,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:26:48,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:26:48,951 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:26:48,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:26:49,010 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['299606 memcached\n'], []) 2021-12-02 07:26:49,012 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 07:26:49,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:26:49,138 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['299606 memcached\n'], []) 2021-12-02 07:26:49,138 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:26:49,142 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:26:49,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:26:49,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:26:49,618 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:26:49,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:26:49,635 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:26:49,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:26:49,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:26:49,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:26:49,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:26:49,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:26:49,742 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:26:49,759 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 07:26:49,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:26:49,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:49,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:49,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:26:49,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:26:49,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:50,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:50,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:26:50,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:26:50,137 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:50,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:50,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:26:50,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:26:50,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:50,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:50,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:26:50,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:26:50,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:26:50,433 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 07:26:50,433 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:26:50,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:50,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:26:50,483 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 07:26:50,483 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:26:50,486 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:26:50,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:26:50,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:26:50,967 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:26:50,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:26:50,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:26:51,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:26:51,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:26:51,038 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:26:51,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:26:51,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:26:51,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:26:51,098 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 07:26:51,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:26:51,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:51,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:51,237 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:26:51,270 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:26:51,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:51,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:51,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:26:51,367 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:26:51,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:26:51,400 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 07:26:51,400 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:26:51,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:51,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:26:51,448 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 07:26:51,448 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:26:51,451 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:26:51,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:26:51,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:26:51,907 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:26:51,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:26:51,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:26:51,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:26:51,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:26:51,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:26:52,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:26:52,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:26:52,043 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:26:52,062 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 07:26:52,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:26:52,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:52,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:52,219 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:26:52,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:26:52,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:52,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:52,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:26:52,443 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:26:52,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:52,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:52,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:26:52,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:26:52,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:52,701 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:52,752 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:26:52,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:26:52,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:52,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:52,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:26:52,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:26:52,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:53,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:53,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:26:53,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:26:53,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:53,130 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:53,163 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:26:53,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:26:53,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:26:53,232 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 07:26:53,232 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:26:53,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:53,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:26:53,299 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 07:26:53,299 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:26:53,302 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:26:53,431 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:26:53,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:26:53,779 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:26:53,780 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:26:53,798 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:26:53,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:26:53,831 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:26:53,862 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:26:53,882 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:26:53,903 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:26:53,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:26:53,944 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 07:26:53,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:26:54,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:54,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:54,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:26:54,132 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:26:54,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:54,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:54,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:26:54,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:26:54,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:54,341 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:54,367 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:26:54,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:26:54,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:54,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:26:54,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:26:54,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:26:54,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:26:54,515 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 07:26:54,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:26:54,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:26:54,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:26:54,561 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 07:26:54,561 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:26:54,561 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:26:54,565 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:26:54,668 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:26:54,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:26:54,996 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:26:54,997 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:26:55,020 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:26:55,040 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:26:55,063 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:26:55,086 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:26:55,104 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:26:55,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:26:55,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:26:55,309 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:26:55,316 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:26:55,316 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:26:57,318 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:26:57,496 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:26:57,502 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:26:57,502 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:26:59,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:26:59,664 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:26:59,670 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:26:59,670 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:01,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:01,844 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:27:01,848 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:01,848 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:03,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:04,028 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:27:04,032 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:04,032 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:06,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:06,207 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:27:06,213 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:06,213 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:08,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:08,391 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:27:08,398 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:08,398 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:10,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:10,526 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:27:10,530 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:10,532 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:12,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:12,694 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:27:12,700 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:12,700 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:14,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:14,842 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:14,842 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:16,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:16,990 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:16,990 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:18,992 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:19,167 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:19,167 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:21,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:21,332 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:21,332 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:23,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:23,477 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:23,479 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:25,479 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:25,631 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:25,631 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:27,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:27,775 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:27,776 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:29,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:29,956 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:29,957 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:31,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:32,177 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:32,178 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:34,178 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:34,384 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:34,385 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:36,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:36,578 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:36,578 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:38,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:38,789 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:38,789 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:40,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:41,013 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:41,013 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:43,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:43,233 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:43,234 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:45,234 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:45,437 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:45,437 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:47,438 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:47,604 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:47,604 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:49,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:49,802 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:49,802 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:51,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:52,006 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:52,007 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:54,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:54,233 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:54,233 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:56,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:56,438 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:56,438 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:27:58,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:27:58,648 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:27:58,650 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:00,650 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:00,861 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:00,861 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:02,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:03,082 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:03,082 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:05,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:05,312 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:05,312 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:07,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:07,499 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:07,500 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:09,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:09,717 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:09,717 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:11,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:11,926 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:11,927 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:13,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:14,098 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:14,098 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:16,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:16,328 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:16,328 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:18,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:18,490 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:18,492 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:20,492 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:20,673 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:20,674 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:22,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:22,832 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:22,832 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:24,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:24,997 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:24,997 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:26,997 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:27,167 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:27,167 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:29,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:29,322 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:29,322 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:31,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:31,484 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:31,486 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:33,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:33,648 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:33,648 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:35,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:35,812 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:35,812 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:37,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:37,976 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:37,976 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:39,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:40,138 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:40,138 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:42,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:42,305 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:42,305 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:44,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:44,510 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:44,510 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:46,512 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:46,719 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:46,720 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:48,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:48,894 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:48,895 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:50,895 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:51,088 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:51,088 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:53,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:53,262 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:53,263 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:55,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:55,451 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:55,451 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:57,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:57,624 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:28:57,625 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:28:59,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:28:59,848 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:28:59,848 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 125.286999941. 2021-12-02 07:28:59,849 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:28:59,851 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:28:59,984 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:29:00,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:29:00,398 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:29:00,400 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:29:00,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:29:00,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:29:00,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:29:00,573 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:29:00,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:29:00,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:29:00,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:29:00,891 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:29:00,891 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.04200005531. 2021-12-02 07:29:00,891 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:29:00,894 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:29:01,012 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:29:01,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:29:01,371 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:29:01,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:29:01,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:29:01,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:29:01,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:29:01,459 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:29:01,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:29:01,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:29:01,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:29:01,700 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:29:01,700 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.809000015259. 2021-12-02 07:29:01,701 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:29:01,703 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:29:01,821 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:29:02,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:29:02,173 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:29:02,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:29:02,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:29:02,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:29:02,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:29:02,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:29:02,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:29:02,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:29:02,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:29:02,440 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:29:02,441 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.740000009537. ############################################################ # # RAM FootPrint: 6045.5078125 # ############################################################ 2021-12-02 07:29:02,480 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 07:29:02,480 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 62 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 2021-12-02 07:30:04,486 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 07:30:04,486 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 07:30:09,490 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:30:09,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:30:09,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:30:09,946 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:30:09,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:30:09,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:30:09,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:30:09,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:30:10,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:30:10,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:30:10,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:30:10,062 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:30:10,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:30:10,117 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['6813 memcached\n'], []) 2021-12-02 07:30:10,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 07:30:10,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:30:10,269 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['6813 memcached\n'], []) 2021-12-02 07:30:10,269 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:30:10,273 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:30:10,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:30:10,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:30:10,732 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:30:10,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:30:10,752 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:30:10,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:30:10,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:30:10,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:30:10,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:30:10,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:30:11,005 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:30:11,005 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:30:11,075 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['93846 memcached\n'], []) 2021-12-02 07:30:11,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 07:30:11,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:30:11,230 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['93846 memcached\n'], []) 2021-12-02 07:30:11,230 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:30:11,236 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:30:11,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:30:11,642 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:30:11,736 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:30:11,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:30:11,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:30:11,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:30:11,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:30:11,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:30:11,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:30:11,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:30:11,887 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:30:11,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:30:11,964 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['218136 memcached\n'], []) 2021-12-02 07:30:11,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 07:30:12,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:30:12,091 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['218136 memcached\n'], []) 2021-12-02 07:30:12,091 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:30:12,094 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:30:14,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:30:15,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:30:15,145 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:30:15,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:30:15,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:30:15,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:30:15,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:30:15,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:30:15,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:30:15,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:30:15,318 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:30:15,318 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:30:15,374 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['300798 memcached\n'], []) 2021-12-02 07:30:15,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 07:30:15,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:30:15,506 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['300798 memcached\n'], []) 2021-12-02 07:30:15,507 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:30:15,510 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:30:15,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:30:15,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:30:15,964 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:30:15,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:30:15,989 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:30:16,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:30:16,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:30:16,056 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:30:16,073 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:30:16,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:30:16,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:30:16,124 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 07:30:16,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:30:16,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:16,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:16,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:30:16,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:30:16,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:16,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:16,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:30:16,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:30:16,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:16,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:16,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:30:16,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:30:16,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:16,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:16,732 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:30:16,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:30:16,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:30:16,798 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 07:30:16,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:30:16,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:16,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:30:16,849 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 07:30:16,849 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:30:16,852 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:30:16,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:30:17,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:30:17,348 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:30:17,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:30:17,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:30:17,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:30:17,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:30:17,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:30:17,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:30:17,456 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:30:17,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:30:17,489 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 07:30:17,522 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:30:17,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:17,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:17,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:30:17,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:30:17,701 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:17,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:17,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:30:17,780 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:30:17,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:30:17,819 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 07:30:17,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:30:17,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:17,854 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:30:17,869 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 07:30:17,871 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:30:17,872 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:30:17,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:30:18,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:30:18,285 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:30:18,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:30:18,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:30:18,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:30:18,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:30:18,362 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:30:18,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:30:18,392 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:30:18,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:30:18,428 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 07:30:18,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:30:18,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:18,522 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:18,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:30:18,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:30:18,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:18,658 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:18,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:30:18,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:30:18,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:18,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:18,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:30:18,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:30:18,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:18,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:18,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:30:18,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:30:18,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:18,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:18,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:30:18,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:30:18,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:19,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:19,045 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:30:19,063 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:30:19,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:19,099 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:19,119 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:30:19,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:30:19,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:30:19,164 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 07:30:19,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:30:19,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:19,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:30:19,203 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 07:30:19,203 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:30:19,206 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:30:19,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:30:19,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:30:19,664 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:30:19,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:30:19,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:30:19,696 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:30:19,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:30:19,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:30:19,752 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:30:19,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:30:19,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:30:19,798 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 07:30:19,823 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:30:19,839 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:19,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:19,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:30:19,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:30:19,938 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:19,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:19,989 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:30:20,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:30:20,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:20,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:20,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:30:20,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:30:20,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:20,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:30:20,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:30:20,150 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:30:20,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:30:20,174 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 07:30:20,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:30:20,186 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:30:20,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:30:20,210 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 07:30:20,210 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:30:20,211 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:30:20,214 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:30:20,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:30:20,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:30:20,674 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:30:20,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:30:20,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:30:20,709 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:30:20,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:30:20,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:30:20,765 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:30:20,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:30:20,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:20,969 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:30:20,974 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:20,976 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:22,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:23,160 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:30:23,165 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:23,165 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:25,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:25,348 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:30:25,352 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:25,352 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:27,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:27,526 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:30:27,530 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:27,530 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:29,530 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:29,704 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:30:29,710 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:29,710 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:31,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:31,887 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:30:31,891 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:31,892 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:33,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:34,056 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:30:34,061 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:34,062 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:36,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:36,239 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:30:36,243 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:36,243 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:38,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:38,348 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:30:38,354 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:38,354 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:40,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:40,540 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:30:40,546 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:40,546 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:42,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:42,694 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:42,694 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:44,696 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:44,849 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:44,849 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:46,849 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:47,006 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:47,006 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:49,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:49,154 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:49,154 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:51,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:51,305 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:51,306 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:53,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:53,451 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:53,453 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:55,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:55,592 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:55,592 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:57,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:57,736 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:57,736 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:30:59,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:30:59,910 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:30:59,910 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:01,910 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:02,134 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:02,135 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:04,137 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:04,299 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:04,299 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:06,301 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:06,516 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:06,516 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:08,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:08,743 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:08,744 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:10,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:10,959 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:10,960 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:12,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:13,117 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:13,118 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:15,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:15,342 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:15,342 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:17,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:17,546 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:17,546 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:19,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:19,726 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:19,726 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:21,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:21,976 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:21,976 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:23,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:24,209 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:24,209 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:26,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:26,430 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:26,430 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:28,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:28,630 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:28,631 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:30,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:30,792 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:30,793 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:32,793 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:33,019 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:33,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:35,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:35,253 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:35,253 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:37,255 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:37,463 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:37,463 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:39,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:39,648 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:39,648 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:41,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:41,864 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:41,865 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:43,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:44,065 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:44,065 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:46,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:46,302 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:46,302 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:48,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:48,467 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:48,467 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:50,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:50,664 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:50,664 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:52,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:52,815 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:52,816 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:54,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:54,980 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:54,980 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:56,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:57,148 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:57,148 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:31:59,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:31:59,332 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:31:59,332 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:01,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:01,513 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:01,513 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:03,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:03,696 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:03,696 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:05,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:05,852 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:05,854 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:07,854 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:08,013 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:08,013 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:10,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:10,184 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:10,184 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:12,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:12,342 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:12,342 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:14,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:14,513 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:14,513 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:16,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:16,663 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:16,663 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:18,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:18,838 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:18,838 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:20,839 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:21,017 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:21,017 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:23,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:23,210 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:23,210 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:25,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:25,394 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:25,394 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:27,395 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:27,585 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:32:27,585 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:32:29,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:29,742 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:32:29,742 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 129.53000021. 2021-12-02 07:32:29,743 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:32:29,744 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:32:29,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:32:31,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:32:32,098 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:32:32,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:32:32,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:32:32,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:32:32,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:32:32,216 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:32:32,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:32:32,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:32:32,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:32,470 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:32:32,470 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 2.72699999809. 2021-12-02 07:32:32,470 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:32:32,473 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:32:32,604 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:32:32,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:32:32,979 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:32:32,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:32:33,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:32:33,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:32:33,043 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:32:33,063 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:32:33,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:32:33,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:32:33,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:33,250 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:32:33,250 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.780999898911. 2021-12-02 07:32:33,253 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:32:33,255 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:32:33,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:32:33,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:32:33,765 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:32:33,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:32:33,785 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:32:33,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:32:33,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:32:33,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:32:33,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:32:33,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:32:34,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:32:34,214 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:32:34,216 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.963000059128. ############################################################ # # RAM FootPrint: 6063.296875 # ############################################################ 2021-12-02 07:32:34,246 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 07:32:34,246 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 111 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 2021-12-02 07:34:25,252 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 07:34:25,252 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 07:34:30,253 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:34:30,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:34:30,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:34:30,733 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:34:30,734 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:34:30,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:34:30,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:34:30,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:34:30,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:34:30,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:34:30,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:34:30,848 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:34:30,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:34:30,911 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['8329 memcached\n'], []) 2021-12-02 07:34:30,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 07:34:30,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:34:31,049 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['8329 memcached\n'], []) 2021-12-02 07:34:31,051 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:34:31,053 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:34:31,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:34:31,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:34:31,525 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:34:31,525 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:34:31,540 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:34:31,553 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:34:31,571 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:34:31,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:34:31,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:34:31,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:34:31,673 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:34:31,673 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:34:31,756 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['94675 memcached\n'], []) 2021-12-02 07:34:31,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 07:34:31,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:34:31,881 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['94675 memcached\n'], []) 2021-12-02 07:34:31,881 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:34:31,884 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:34:32,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:34:32,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:34:32,391 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:34:32,392 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:34:32,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:34:32,433 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:34:32,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:34:32,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:34:32,499 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:34:32,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:34:32,536 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:34:32,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:34:32,631 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['218936 memcached\n'], []) 2021-12-02 07:34:32,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 07:34:32,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:34:32,785 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['218936 memcached\n'], []) 2021-12-02 07:34:32,785 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:34:32,788 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:34:32,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:34:33,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:34:33,246 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:34:33,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:34:33,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:34:33,288 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:34:33,313 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:34:33,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:34:33,359 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:34:33,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:34:33,398 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:34:33,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:34:33,464 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['301847 memcached\n'], []) 2021-12-02 07:34:33,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 07:34:33,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:34:33,596 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['301847 memcached\n'], []) 2021-12-02 07:34:33,596 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:34:33,599 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:34:33,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:34:33,997 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:34:34,078 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:34:34,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:34:34,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:34:34,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:34:34,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:34:34,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:34:34,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:34:34,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:34:34,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:34:34,221 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 07:34:34,252 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:34:34,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:34,319 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:34,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:34:34,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:34:34,431 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:34,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:34,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:34:34,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:34:34,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:34,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:34,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:34:34,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:34:34,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:34,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:34,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:34:34,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:34:34,775 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:34:34,786 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 07:34:34,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:34:34,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:34,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:34:34,822 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 07:34:34,822 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:34:34,825 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:34:34,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:34:35,239 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:34:35,315 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:34:35,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:34:35,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:34:35,349 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:34:35,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:34:35,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:34:35,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:34:35,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:34:35,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:34:35,451 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 07:34:35,479 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:34:35,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:35,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:35,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:34:35,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:34:35,591 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:35,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:35,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:34:35,658 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:34:35,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:34:35,687 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 07:34:35,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:34:35,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:35,714 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:34:35,727 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 07:34:35,727 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:34:35,730 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:34:35,859 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:34:36,119 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:34:36,197 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:34:36,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:34:36,216 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:34:36,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:34:36,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:34:36,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:34:36,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:34:36,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:34:36,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:34:36,339 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 07:34:36,367 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:34:36,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:36,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:36,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:34:36,525 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:34:36,556 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:36,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:36,655 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:34:36,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:34:36,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:36,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:36,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:34:36,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:34:36,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:36,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:36,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:34:36,982 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:34:37,010 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:37,043 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:37,073 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:34:37,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:34:37,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:37,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:37,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:34:37,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:34:37,234 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:37,260 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:37,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:34:37,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:34:37,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:37,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:37,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:34:37,367 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:34:37,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:34:37,392 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 07:34:37,392 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:34:37,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:37,418 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:34:37,433 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 07:34:37,433 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:34:37,436 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:34:37,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:34:37,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:34:37,897 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:34:37,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:34:37,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:34:37,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:34:37,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:34:37,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:34:37,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:34:38,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:34:38,030 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:34:38,048 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 07:34:38,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:34:38,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:38,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:38,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:34:38,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:34:38,265 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:38,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:38,362 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:34:38,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:34:38,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:38,482 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:38,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:34:38,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:34:38,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:38,630 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:34:38,658 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:34:38,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:34:38,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:34:38,719 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 07:34:38,719 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:34:38,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:34:38,755 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:34:38,769 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 07:34:38,770 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:34:38,770 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:34:38,773 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:34:38,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:34:39,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:34:39,246 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:34:39,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:34:39,265 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:34:39,280 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:34:39,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:34:39,319 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:34:39,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:34:39,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:34:39,362 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:34:40,562 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-02 07:34:40,566 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:34:40,566 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:34:42,568 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:34:42,737 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:34:42,744 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:34:42,746 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:34:44,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:34:44,927 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:34:44,933 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:34:44,933 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:34:46,933 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:34:47,111 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:34:47,115 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:34:47,115 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:34:49,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:34:49,279 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:34:49,285 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:34:49,285 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:34:51,285 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:34:51,460 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:34:51,467 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:34:51,467 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:34:53,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:34:53,632 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:34:53,638 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:34:53,638 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:34:55,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:34:55,815 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:34:55,819 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:34:55,819 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:34:57,821 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:34:57,997 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:34:58,003 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:34:58,003 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:00,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:00,167 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:35:00,171 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:00,171 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:02,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:02,315 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:02,316 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:04,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:04,463 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:04,464 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:06,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:06,611 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:06,611 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:08,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:08,763 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:08,763 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:10,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:10,920 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:10,920 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:12,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:13,061 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:13,062 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:15,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:15,203 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:15,203 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:17,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:17,414 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:17,414 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:19,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:19,573 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:19,573 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:21,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:21,773 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:21,773 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:23,775 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:24,006 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:24,006 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:26,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:26,240 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:26,240 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:28,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:28,438 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:28,440 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:30,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:30,658 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:30,658 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:32,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:32,832 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:32,832 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:34,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:35,055 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:35,055 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:37,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:37,207 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:37,207 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:39,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:39,448 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:39,450 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:41,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:41,676 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:41,676 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:43,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:43,900 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:43,901 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:45,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:46,122 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:46,122 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:48,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:48,345 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:48,345 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:50,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:50,552 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:50,553 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:52,553 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:52,778 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:52,779 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:54,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:54,941 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:54,941 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:56,943 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:57,105 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:57,105 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:35:59,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:35:59,269 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:35:59,269 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:01,270 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:01,434 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:01,436 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:03,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:03,605 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:03,605 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:05,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:05,793 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:05,793 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:07,793 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:07,963 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:07,963 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:09,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:10,155 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:10,157 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:12,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:12,315 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:12,315 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:14,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:14,476 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:14,476 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:16,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:16,625 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:16,625 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:18,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:18,786 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:18,786 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:20,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:20,940 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:20,940 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:22,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:23,132 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:23,134 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:25,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:25,338 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:25,338 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:27,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:27,512 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:27,512 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:29,512 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:29,683 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:29,683 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:31,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:31,865 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:31,865 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:33,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:34,039 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:34,039 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:36,040 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:36,207 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:36,207 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:38,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:38,385 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:36:38,385 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:36:40,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:40,630 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:36:40,631 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 121.859000206. 2021-12-02 07:36:40,631 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:36:40,634 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:36:40,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:36:41,102 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:36:41,193 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:36:41,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:36:41,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:36:41,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:36:41,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:36:41,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:36:41,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:36:41,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:36:41,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:41,556 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:36:41,558 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.927000045776. 2021-12-02 07:36:41,559 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:36:41,559 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:36:41,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:36:41,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:36:42,065 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:36:42,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:36:42,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:36:42,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:36:42,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:36:42,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:36:42,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:36:42,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:36:42,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:42,427 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:36:42,427 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.868000030518. 2021-12-02 07:36:42,427 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:36:42,430 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:36:42,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:36:42,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:36:42,869 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:36:42,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:36:42,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:36:42,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:36:42,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:36:42,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:36:42,954 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:36:42,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:36:42,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:36:43,115 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:36:43,115 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.68799996376. ############################################################ # # RAM FootPrint: 6102.15625 # ############################################################ 2021-12-02 07:36:43,147 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 07:36:43,147 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 68 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 2021-12-02 07:37:51,151 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 07:37:51,151 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 07:37:56,154 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:37:56,301 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:37:56,591 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:37:56,713 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:37:56,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:37:56,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:37:56,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:37:56,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:37:56,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:37:57,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:37:57,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:37:57,062 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:37:57,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:37:57,141 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['10012 memcached\n'], []) 2021-12-02 07:37:57,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 07:37:57,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:37:57,290 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['10012 memcached\n'], []) 2021-12-02 07:37:57,292 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:37:57,296 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:37:57,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:37:57,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:37:57,819 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:37:57,821 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:37:57,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:37:57,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:37:57,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:37:57,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:37:57,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:37:58,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:37:58,056 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:37:58,056 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:37:58,164 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['95577 memcached\n'], []) 2021-12-02 07:37:58,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 07:37:58,290 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:37:58,382 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['95577 memcached\n'], []) 2021-12-02 07:37:58,382 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:37:58,387 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:37:58,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:37:58,714 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:37:58,779 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:37:58,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:37:58,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:37:58,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:37:58,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:37:58,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:37:58,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:37:58,872 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:37:58,894 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:37:58,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:37:58,937 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['219792 memcached\n'], []) 2021-12-02 07:37:58,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 07:37:58,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:37:59,042 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['219792 memcached\n'], []) 2021-12-02 07:37:59,042 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:37:59,045 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:37:59,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:37:59,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:37:59,529 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:37:59,530 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:37:59,553 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:37:59,573 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:37:59,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:37:59,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:37:59,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:37:59,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:37:59,676 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:37:59,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:37:59,729 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['303014 memcached\n'], []) 2021-12-02 07:37:59,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 07:37:59,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:37:59,881 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['303014 memcached\n'], []) 2021-12-02 07:37:59,881 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:37:59,884 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:37:59,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:38:00,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:38:00,335 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:38:00,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:38:00,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:38:00,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:38:00,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:38:00,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:38:00,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:38:00,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:38:00,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:38:00,463 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 07:38:00,489 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:38:00,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:00,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:00,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:38:00,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:38:00,658 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:00,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:00,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:38:00,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:38:00,831 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:00,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:00,926 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:38:00,959 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:38:00,987 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:01,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:01,058 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:38:01,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:38:01,109 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:38:01,125 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 07:38:01,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:38:01,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:01,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:38:01,171 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 07:38:01,171 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:38:01,174 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:38:01,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:38:01,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:38:01,676 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:38:01,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:38:01,696 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:38:01,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:38:01,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:38:01,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:38:01,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:38:01,785 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:38:01,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:38:01,819 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 07:38:01,846 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:38:01,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:01,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:01,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:38:01,992 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:38:02,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:02,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:02,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:38:02,119 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:38:02,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:38:02,161 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 07:38:02,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:38:02,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:02,201 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:38:02,217 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 07:38:02,217 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:38:02,221 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:38:02,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:38:02,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:38:02,671 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:38:02,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:38:02,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:38:02,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:38:02,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:38:02,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:38:02,778 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:38:02,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:38:02,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:38:02,834 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 07:38:02,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:38:02,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:02,938 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:02,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:38:03,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:38:03,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:03,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:03,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:38:03,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:38:03,242 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:03,290 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:03,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:38:03,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:38:03,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:03,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:03,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:38:03,548 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:38:03,581 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:03,622 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:03,665 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:38:03,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:38:03,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:03,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:03,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:38:03,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:38:03,877 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:03,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:03,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:38:04,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:38:04,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:04,063 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:04,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:38:04,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:38:04,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:38:04,157 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 07:38:04,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:38:04,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:04,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:38:04,216 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 07:38:04,216 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:38:04,219 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:38:04,323 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:38:04,602 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:38:04,681 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:38:04,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:38:04,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:38:04,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:38:04,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:38:04,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:38:04,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:38:04,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:38:04,785 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:38:04,799 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 07:38:04,826 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:38:04,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:04,872 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:04,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:38:04,926 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:38:04,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:04,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:05,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:38:05,061 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:38:05,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:05,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:05,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:38:05,191 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:38:05,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:05,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:38:05,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:38:05,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:38:05,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:38:05,312 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 07:38:05,313 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:38:05,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:38:05,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:38:05,357 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 07:38:05,357 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:38:05,357 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:38:05,359 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:38:05,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:38:07,392 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:38:07,483 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:38:07,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:38:07,505 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:38:07,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:38:07,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:38:07,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:38:07,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:38:07,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:38:07,619 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:07,765 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:38:07,772 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:07,772 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:09,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:09,934 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:38:09,938 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:09,938 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:11,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:12,102 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:38:12,108 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:12,108 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:14,109 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:14,295 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:38:14,299 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:14,299 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:16,301 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:16,464 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:38:16,470 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:16,470 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:18,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:18,634 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:38:18,638 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:18,638 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:20,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:20,753 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:38:20,759 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:20,759 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:22,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:22,882 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:38:22,888 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:22,888 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:24,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:25,032 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:25,033 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:27,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:27,180 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:27,180 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:29,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:29,358 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:29,358 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:31,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:31,512 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:31,512 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:33,512 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:33,654 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:33,654 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:35,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:35,796 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:35,796 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:37,798 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:37,940 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:37,940 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:39,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:40,137 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:40,137 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:42,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:42,357 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:42,357 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:44,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:44,545 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:44,545 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:46,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:46,697 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:46,697 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:48,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:48,897 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:48,897 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:50,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:51,088 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:51,088 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:53,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:53,253 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:53,253 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:55,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:55,415 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:55,415 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:57,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:57,654 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:57,654 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:38:59,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:38:59,882 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:38:59,882 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:01,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:02,091 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:02,091 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:04,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:04,312 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:04,312 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:06,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:06,499 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:06,499 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:08,499 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:08,704 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:08,706 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:10,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:10,901 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:10,901 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:12,903 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:13,118 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:13,118 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:15,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:15,329 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:15,329 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:17,329 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:17,483 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:17,483 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:19,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:19,660 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:19,660 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:21,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:21,822 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:21,822 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:23,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:23,979 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:23,979 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:25,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:26,141 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:26,141 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:28,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:28,290 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:28,290 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:30,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:30,459 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:30,460 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:32,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:32,625 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:32,625 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:34,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:34,790 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:34,790 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:36,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:36,980 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:36,980 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:38,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:39,158 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:39,158 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:41,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:41,342 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:41,342 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:43,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:43,493 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:43,493 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:45,494 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:45,671 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:45,671 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:47,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:47,846 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:47,846 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:49,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:50,029 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:50,029 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:52,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:52,201 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:52,201 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:54,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:54,378 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:54,378 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:56,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:56,530 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:56,532 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:39:58,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:39:58,704 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:39:58,704 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:40:00,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:40:00,874 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:40:00,874 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:40:02,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:40:03,042 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:40:03,043 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 117.686999798. 2021-12-02 07:40:03,045 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:40:03,048 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:40:03,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:40:03,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:40:03,654 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:40:03,655 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:40:03,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:40:03,721 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:40:03,836 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:40:03,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:40:03,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:40:03,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:40:03,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:40:04,147 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:40:04,147 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.10199999809. 2021-12-02 07:40:04,147 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:40:04,150 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:40:04,260 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:40:04,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:40:04,638 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:40:04,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:40:04,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:40:04,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:40:04,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:40:04,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:40:04,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:40:04,762 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:40:04,780 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:40:04,937 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:40:04,937 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.789999961853. 2021-12-02 07:40:04,937 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:40:04,940 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:40:05,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:40:05,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:40:05,328 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:40:05,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:40:05,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:40:05,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:40:05,400 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:40:05,426 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:40:05,448 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:40:05,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:40:05,489 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:40:05,696 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:40:05,697 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.759000062943. ############################################################ # # RAM FootPrint: 6109.01171875 # ############################################################ 2021-12-02 07:40:05,729 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 07:40:05,729 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 88 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 2021-12-02 07:41:33,734 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 07:41:33,734 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 07:41:38,736 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:41:39,295 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:41:39,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:41:39,884 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:41:39,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:41:39,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:41:39,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:41:39,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:41:39,966 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:41:39,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:41:39,997 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:41:40,015 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:41:40,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:41:40,095 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['11441 memcached\n'], []) 2021-12-02 07:41:40,096 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 07:41:40,224 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:41:40,293 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['11441 memcached\n'], []) 2021-12-02 07:41:40,293 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:41:40,296 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:41:40,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:41:40,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:41:40,845 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:41:40,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:41:40,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:41:40,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:41:40,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:41:40,900 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:41:40,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:41:40,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:41:40,977 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:41:40,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:41:41,036 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['96409 memcached\n'], []) 2021-12-02 07:41:41,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 07:41:41,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:41:41,161 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['96409 memcached\n'], []) 2021-12-02 07:41:41,161 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:41:41,164 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:41:41,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:41:41,551 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:41:41,638 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:41:41,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:41:41,658 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:41:41,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:41:41,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:41:41,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:41:41,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:41:41,765 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:41:41,786 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:41:41,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:41:41,864 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['220589 memcached\n'], []) 2021-12-02 07:41:41,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 07:41:41,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:41:41,989 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['220589 memcached\n'], []) 2021-12-02 07:41:41,990 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:41:41,993 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:41:42,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:41:42,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:41:42,456 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:41:42,456 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:41:42,482 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:41:42,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:41:42,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:41:42,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:41:42,571 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:41:42,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:41:42,607 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:41:42,607 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:41:42,673 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['1438 memcached\n'], []) 2021-12-02 07:41:42,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 07:41:42,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:41:42,826 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['1438 memcached\n'], []) 2021-12-02 07:41:42,826 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:41:42,829 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:41:42,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:41:43,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:41:43,286 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:41:43,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:41:43,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:41:43,323 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:41:43,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:41:43,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:41:43,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:41:43,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:41:43,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:41:43,427 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 07:41:43,454 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:41:43,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:43,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:43,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:41:43,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:41:43,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:43,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:43,742 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:41:43,780 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:41:43,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:43,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:43,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:41:43,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:41:43,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:44,022 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:44,061 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:41:44,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:41:44,114 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:41:44,131 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 07:41:44,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:41:44,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:44,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:41:44,186 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 07:41:44,186 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:41:44,188 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:41:44,301 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:41:44,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:41:44,895 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:41:44,897 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:41:44,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:41:44,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:41:44,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:41:44,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:41:44,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:41:45,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:41:45,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:41:45,038 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 07:41:45,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:41:45,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:45,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:45,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:41:45,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:41:45,239 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:45,272 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:45,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:41:45,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:41:45,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:41:45,368 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 07:41:45,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:41:45,382 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:45,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:41:45,411 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 07:41:45,411 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:41:45,414 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:41:45,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:41:45,780 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:41:45,854 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:41:45,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:41:45,872 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:41:45,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:41:45,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:41:45,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:41:45,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:41:45,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:41:45,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:41:45,993 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 07:41:46,020 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:41:46,043 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:46,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:46,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:41:46,186 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:41:46,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:46,272 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:46,329 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:41:46,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:41:46,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:46,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:46,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:41:46,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:41:46,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:46,642 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:46,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:41:46,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:41:46,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:46,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:46,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:41:46,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:41:46,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:46,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:47,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:41:47,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:41:47,076 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:47,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:47,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:41:47,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:41:47,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:47,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:47,272 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:41:47,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:41:47,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:41:47,328 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 07:41:47,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:41:47,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:47,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:41:47,378 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 07:41:47,380 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:41:47,382 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:41:47,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:41:47,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:41:47,868 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:41:47,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:41:47,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:41:47,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:41:47,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:41:47,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:41:47,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:41:47,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:41:48,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:41:48,016 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 07:41:48,043 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:41:48,063 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:48,109 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:48,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:41:48,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:41:48,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:48,232 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:48,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:41:48,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:41:48,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:48,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:48,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:41:48,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:41:48,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:48,469 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:41:48,505 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:41:48,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:41:48,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:41:48,575 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 07:41:48,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:41:48,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:41:48,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:41:48,628 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 07:41:48,628 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:41:48,628 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:41:48,631 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:41:48,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:41:49,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:41:49,098 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:41:49,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:41:49,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:41:49,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:41:49,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:41:49,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:41:49,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:41:49,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:41:49,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:41:49,407 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-02 07:41:49,411 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:41:49,413 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:41:51,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:41:51,582 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:41:51,586 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:41:51,586 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:41:53,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:41:53,704 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:41:53,710 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:41:53,710 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:41:55,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:41:55,881 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:41:55,888 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:41:55,888 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:41:57,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:41:58,059 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:41:58,066 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:41:58,066 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:00,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:00,233 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:42:00,240 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:00,240 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:02,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:02,403 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:42:02,407 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:02,407 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:04,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:04,588 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:42:04,594 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:04,594 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:06,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:06,755 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:42:06,759 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:06,759 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:08,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:08,931 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:42:08,937 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:08,937 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:10,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:11,082 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:11,082 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:13,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:13,246 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:13,246 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:15,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:15,401 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:15,401 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:17,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:17,546 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:17,546 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:19,548 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:19,697 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:19,697 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:21,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:21,842 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:21,844 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:23,844 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:24,003 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:24,003 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:26,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:26,142 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:26,144 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:28,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:28,354 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:28,354 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:30,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:30,525 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:30,525 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:32,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:32,724 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:32,726 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:34,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:34,944 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:34,944 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:36,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:37,157 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:37,157 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:39,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:39,352 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:39,352 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:41,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:41,566 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:41,566 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:43,568 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:43,786 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:43,786 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:45,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:46,015 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:46,015 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:48,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:48,223 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:48,223 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:50,224 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:50,427 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:50,427 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:52,428 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:52,638 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:52,640 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:54,640 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:54,821 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:54,822 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:56,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:56,980 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:56,980 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:42:58,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:42:59,141 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:42:59,141 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:01,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:01,299 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:01,299 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:03,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:03,460 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:03,460 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:05,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:05,631 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:05,631 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:07,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:07,782 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:07,782 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:09,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:09,980 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:09,982 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:11,982 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:12,191 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:12,193 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:14,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:14,374 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:14,375 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:16,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:16,546 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:16,546 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:18,548 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:18,714 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:18,714 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:20,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:20,878 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:20,878 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:22,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:23,032 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:23,032 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:25,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:25,196 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:25,196 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:27,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:27,362 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:27,362 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:29,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:29,533 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:29,533 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:31,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:31,697 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:31,697 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:33,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:33,864 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:33,864 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:35,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:36,026 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:36,026 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:38,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:38,197 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:38,197 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:40,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:40,374 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:43:40,374 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:43:42,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:42,582 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:43:42,582 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 113.953999996. 2021-12-02 07:43:42,582 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:43:42,586 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:43:42,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:43:43,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:43:43,108 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:43:43,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:43:43,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:43:43,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:43:43,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:43:43,260 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:43:43,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:43:43,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:43:43,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:43,512 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:43:43,512 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.929000139236. 2021-12-02 07:43:43,513 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:43:43,515 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:43:43,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:43:43,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:43:44,016 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:43:44,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:43:44,038 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:43:44,058 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:43:44,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:43:44,102 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:43:44,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:43:44,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:43:44,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:44,306 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:43:44,308 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.795000076294. 2021-12-02 07:43:44,309 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:43:44,309 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:43:44,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:43:44,630 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:43:44,714 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:43:44,714 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:43:44,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:43:44,749 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:43:44,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:43:44,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:43:44,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:43:44,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:43:44,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:43:45,038 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:43:45,039 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.730000019073. ############################################################ # # RAM FootPrint: 6049.68359375 # ############################################################ 2021-12-02 07:43:45,072 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 07:43:45,072 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 90 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 2021-12-02 07:45:15,076 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 07:45:15,076 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 07:45:20,079 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:45:20,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:45:20,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:45:20,595 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:45:20,596 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:45:20,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:45:20,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:45:20,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:45:20,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:45:20,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:45:20,719 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:45:20,823 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:45:20,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:45:20,885 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['13012 memcached\n'], []) 2021-12-02 07:45:20,885 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 07:45:20,971 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:45:21,105 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['13012 memcached\n'], []) 2021-12-02 07:45:21,105 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:45:21,108 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:45:21,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:45:21,617 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:45:21,736 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:45:21,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:45:21,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:45:21,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:45:21,844 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:45:21,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:45:21,887 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:45:21,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:45:21,947 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:45:21,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:45:22,009 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['97262 memcached\n'], []) 2021-12-02 07:45:22,010 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 07:45:22,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:45:22,217 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['97262 memcached\n'], []) 2021-12-02 07:45:22,217 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:45:22,220 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:45:22,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:45:22,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:45:22,723 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:45:22,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:45:22,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:45:22,765 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:45:22,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:45:22,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:45:22,831 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:45:22,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:45:22,869 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:45:22,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:45:22,947 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['221411 memcached\n'], []) 2021-12-02 07:45:22,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 07:45:23,005 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:45:23,071 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['221411 memcached\n'], []) 2021-12-02 07:45:23,071 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:45:23,073 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:45:23,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:45:23,540 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:45:23,645 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:45:23,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:45:23,665 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:45:23,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:45:23,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:45:23,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:45:23,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:45:23,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:45:23,864 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:45:23,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:45:23,927 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['2629 memcached\n'], []) 2021-12-02 07:45:23,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 07:45:24,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:45:24,135 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['2629 memcached\n'], []) 2021-12-02 07:45:24,135 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:45:24,138 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:45:24,247 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:45:24,517 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:45:24,598 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:45:24,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:45:24,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:45:24,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:45:24,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:45:24,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:45:24,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:45:24,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:45:24,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:45:24,713 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 07:45:24,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:45:24,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:24,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:24,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:45:24,887 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:45:24,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:24,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:25,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:45:25,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:45:25,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:25,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:25,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:45:25,183 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:45:25,206 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:25,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:25,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:45:25,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:45:25,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:25,318 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:25,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:45:25,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:45:25,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:45:25,372 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 07:45:25,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:45:25,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:25,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:45:25,414 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 07:45:25,414 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:45:25,417 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:45:25,528 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:45:28,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:45:28,993 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:45:28,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:45:29,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:45:29,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:45:29,053 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:45:29,073 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:45:29,091 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:45:29,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:45:29,122 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:45:29,138 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 07:45:29,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:45:29,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:29,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:29,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:45:29,319 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:45:29,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:29,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:29,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:45:29,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:45:29,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:45:29,513 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 07:45:29,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:45:29,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:29,551 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:45:29,568 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 07:45:29,568 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:45:29,572 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:45:29,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:45:29,946 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:45:30,028 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:45:30,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:45:30,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:45:30,068 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:45:30,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:45:30,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:45:30,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:45:30,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:45:30,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:45:30,177 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 07:45:30,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:45:30,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:30,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:30,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:45:30,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:45:30,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:30,390 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:30,418 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:45:30,441 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:45:30,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:30,489 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:30,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:45:30,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:45:30,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:30,573 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:30,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:45:30,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:45:30,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:30,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:30,673 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:45:30,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:45:30,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:30,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:30,747 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:45:30,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:45:30,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:30,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:30,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:45:30,846 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:45:30,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:30,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:30,903 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:45:30,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:45:30,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:45:30,947 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 07:45:30,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:45:30,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:30,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:45:30,987 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 07:45:30,987 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:45:30,990 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:45:31,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:45:31,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:45:31,523 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:45:31,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:45:31,545 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:45:31,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:45:31,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:45:31,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:45:31,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:45:31,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:45:31,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:45:31,703 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 07:45:31,732 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:45:31,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:31,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:31,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:45:31,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:45:31,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:31,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:32,020 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:45:32,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:45:32,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:32,127 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:32,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:45:32,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:45:32,237 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:32,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:45:32,303 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:45:32,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:45:32,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:45:32,375 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 07:45:32,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:45:32,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:45:32,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:45:32,431 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 07:45:32,433 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:45:32,433 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:45:32,436 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:45:32,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:45:32,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:45:32,907 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:45:32,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:45:32,926 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:45:32,943 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:45:32,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:45:32,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:45:33,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:45:33,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:45:33,043 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:33,163 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:45:33,167 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:33,167 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:45:35,168 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:35,326 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:45:35,334 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:35,334 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:45:37,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:37,492 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:45:37,496 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:37,496 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:45:39,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:39,678 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:45:39,684 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:39,684 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:45:41,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:41,861 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:45:41,868 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:41,868 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:45:43,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:44,028 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:45:44,032 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:44,032 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:45:46,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:46,176 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:45:46,180 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:46,180 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:45:48,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:48,328 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:45:48,334 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:48,334 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:45:50,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:50,479 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:50,479 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:45:52,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:52,628 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:52,630 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:45:54,630 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:54,773 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:54,773 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:45:56,775 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:56,918 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:56,920 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:45:58,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:45:59,062 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:45:59,062 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:01,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:01,217 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:01,217 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:03,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:03,359 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:03,359 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:05,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:05,558 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:05,559 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:07,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:07,756 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:07,757 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:09,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:09,971 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:09,973 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:11,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:12,173 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:12,173 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:14,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:14,404 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:14,404 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:16,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:16,608 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:16,608 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:18,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:18,825 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:18,826 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:20,826 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:21,033 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:21,035 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:23,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:23,266 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:23,266 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:25,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:25,493 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:25,493 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:27,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:27,703 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:27,703 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:29,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:29,888 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:29,888 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:31,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:32,111 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:32,111 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:34,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:34,325 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:34,325 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:36,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:36,576 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:36,578 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:38,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:38,729 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:38,729 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:40,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:40,890 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:40,890 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:42,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:43,045 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:43,045 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:45,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:45,194 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:45,194 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:47,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:47,349 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:47,349 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:49,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:49,533 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:49,533 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:51,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:51,711 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:51,711 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:53,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:53,878 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:53,878 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:55,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:56,042 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:56,042 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:46:58,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:46:58,197 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:46:58,197 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:47:00,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:00,359 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:47:00,359 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:47:02,359 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:02,528 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:47:02,528 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:47:04,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:04,694 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:47:04,694 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:47:06,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:06,846 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:47:06,848 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:47:08,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:09,019 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:47:09,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:47:11,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:11,180 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:47:11,181 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:47:13,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:13,346 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:47:13,346 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:47:15,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:15,507 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:47:15,507 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:47:17,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:17,696 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:47:17,696 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:47:19,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:19,871 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:47:19,871 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:47:21,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:22,036 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:47:22,038 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:47:24,038 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:24,244 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:47:24,246 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 111.812999964. 2021-12-02 07:47:24,246 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:47:24,249 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:47:24,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:47:24,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:47:24,779 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:47:24,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:47:24,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:47:24,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:47:24,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:47:24,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:47:24,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:47:24,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:47:25,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:25,198 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:47:25,200 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.953000068665. 2021-12-02 07:47:25,200 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:47:25,203 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:47:25,326 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:47:25,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:47:25,713 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:47:25,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:47:25,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:47:25,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:47:25,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:47:25,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:47:25,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:47:25,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:47:25,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:26,055 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:47:26,055 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.854000091553. 2021-12-02 07:47:26,055 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:47:26,058 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:47:26,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:47:26,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:47:26,510 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:47:26,512 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:47:26,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:47:26,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:47:26,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:47:26,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:47:26,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:47:26,630 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:47:26,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:47:26,802 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:47:26,802 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.74799990654. ############################################################ # # RAM FootPrint: 6039.9453125 # ############################################################ 2021-12-02 07:47:26,832 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 07:47:26,832 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 84 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 2021-12-02 07:48:50,838 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 07:48:50,838 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 07:48:55,842 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:48:55,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:48:58,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:48:58,888 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:48:58,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:48:58,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:48:58,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:48:58,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:48:58,989 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:48:59,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:48:59,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:48:59,036 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:48:59,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:48:59,092 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['14452 memcached\n'], []) 2021-12-02 07:48:59,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 07:48:59,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:48:59,283 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['14452 memcached\n'], []) 2021-12-02 07:48:59,285 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:48:59,286 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:48:59,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:48:59,747 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:48:59,855 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:48:59,857 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:49:00,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:49:00,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:49:00,056 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:49:00,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:49:00,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:49:00,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:49:00,196 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:49:00,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:49:00,334 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['97902 memcached\n'], []) 2021-12-02 07:49:00,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 07:49:00,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:49:00,473 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['97902 memcached\n'], []) 2021-12-02 07:49:00,473 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:49:00,476 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:49:00,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:49:00,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:49:00,967 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:49:00,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:49:00,989 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:49:01,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:49:01,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:49:01,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:49:01,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:49:01,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:49:01,114 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:49:01,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:49:01,188 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['222225 memcached\n'], []) 2021-12-02 07:49:01,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 07:49:01,244 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:49:01,316 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['222225 memcached\n'], []) 2021-12-02 07:49:01,316 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:49:01,319 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:49:01,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:49:01,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:49:01,733 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:49:01,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:49:01,755 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:49:01,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:49:01,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:49:01,823 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:49:01,844 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:49:01,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:49:01,884 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:49:01,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:49:01,933 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['3814 memcached\n'], []) 2021-12-02 07:49:01,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 07:49:02,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:49:02,042 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['3814 memcached\n'], []) 2021-12-02 07:49:02,042 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:49:02,046 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:49:02,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:49:02,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:49:02,509 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:49:02,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:49:02,530 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:49:02,551 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:49:02,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:49:02,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:49:02,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:49:02,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:49:02,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:49:02,661 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 07:49:02,691 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:49:02,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:02,762 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:02,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:49:02,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:49:02,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:02,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:02,900 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:49:02,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:49:02,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:02,966 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:02,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:49:03,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:49:03,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:03,073 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:03,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:49:03,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:49:03,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:03,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:03,206 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:49:03,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:49:03,237 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:49:03,253 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 07:49:03,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:49:03,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:03,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:49:03,296 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 07:49:03,296 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:49:03,299 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:49:03,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:49:04,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:49:04,137 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:49:04,137 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:49:04,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:49:04,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:49:04,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:49:04,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:49:04,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:49:04,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:49:04,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:49:04,283 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 07:49:04,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:49:04,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:04,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:04,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:49:04,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:49:04,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:04,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:04,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:49:04,604 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:49:04,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:04,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:04,673 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:49:04,691 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:49:04,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:49:04,726 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 07:49:04,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:49:04,742 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:04,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:49:04,773 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 07:49:04,773 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:49:04,776 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:49:04,882 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:49:05,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:49:05,230 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:49:05,232 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:49:05,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:49:05,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:49:05,295 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:49:05,318 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:49:05,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:49:05,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:49:05,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:49:05,388 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 07:49:05,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:49:05,441 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:05,489 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:05,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:49:05,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:49:05,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:05,668 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:05,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:49:05,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:49:05,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:05,836 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:05,887 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:49:05,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:49:05,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:05,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:06,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:49:06,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:49:06,096 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:06,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:06,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:49:06,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:49:06,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:06,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:06,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:49:06,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:49:06,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:06,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:06,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:49:06,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:49:06,456 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:06,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:06,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:49:06,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:49:06,553 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:49:06,571 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 07:49:06,571 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:49:06,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:06,604 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:49:06,618 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 07:49:06,618 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:49:06,621 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:49:06,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:49:06,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:49:07,069 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:49:07,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:49:07,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:49:07,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:49:07,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:49:07,132 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:49:07,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:49:07,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:49:07,170 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:49:07,184 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 07:49:07,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:49:07,227 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:07,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:07,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:49:07,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:49:07,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:07,418 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:07,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:49:07,507 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:49:07,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:07,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:07,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:49:07,647 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:49:07,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:07,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:49:07,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:49:07,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:49:07,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:49:07,809 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 07:49:07,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:49:07,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:49:07,844 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:49:07,858 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 07:49:07,858 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:49:07,858 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:49:07,861 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:49:07,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:49:09,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:49:09,092 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:49:09,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:49:09,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:49:09,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:49:09,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:49:09,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:49:09,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:49:09,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:49:09,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:09,398 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:49:09,405 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:09,405 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:11,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:11,572 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:49:11,579 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:11,579 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:13,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:13,762 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:49:13,766 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:13,766 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:15,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:15,944 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:49:15,950 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:15,950 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:17,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:18,125 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:49:18,130 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:18,130 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:20,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:20,303 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:49:20,309 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:20,309 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:22,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:22,480 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:49:22,483 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:22,483 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:24,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:24,657 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:49:24,661 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:24,661 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:26,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:26,819 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:26,819 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:28,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:28,961 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:28,961 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:30,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:31,127 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:31,127 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:33,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:33,276 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:33,276 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:35,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:35,418 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:35,418 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:37,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:37,573 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:37,573 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:39,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:39,726 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:39,726 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:41,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:41,931 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:41,931 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:43,933 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:44,095 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:44,095 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:46,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:46,325 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:46,325 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:48,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:48,493 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:48,493 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:50,494 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:50,726 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:50,726 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:52,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:52,910 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:52,910 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:54,910 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:55,151 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:55,151 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:57,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:57,387 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:57,387 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:49:59,387 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:49:59,618 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:49:59,618 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:01,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:01,846 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:01,848 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:03,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:04,062 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:04,062 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:06,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:06,269 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:06,269 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:08,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:08,470 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:08,470 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:10,471 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:10,691 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:10,691 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:12,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:12,859 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:12,859 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:14,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:15,026 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:15,026 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:17,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:17,187 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:17,187 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:19,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:19,342 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:19,344 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:21,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:21,507 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:21,509 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:23,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:23,674 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:23,674 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:25,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:25,857 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:25,857 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:27,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:28,025 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:28,025 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:30,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:30,190 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:30,191 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:32,191 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:32,388 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:32,388 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:34,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:34,565 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:34,566 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:36,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:36,732 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:36,732 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:38,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:38,897 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:38,897 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:40,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:41,058 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:41,059 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:43,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:43,233 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:43,233 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:45,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:45,407 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:45,408 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:47,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:47,588 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:47,588 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:49,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:49,757 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:49,759 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:51,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:51,933 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:51,933 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:53,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:54,107 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:54,107 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:56,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:56,273 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:50:56,273 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:50:58,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:58,443 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:50:58,444 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 110.585999966. 2021-12-02 07:50:58,444 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:50:58,446 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:50:58,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:50:58,887 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:50:58,974 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:50:58,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:50:58,992 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:50:59,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:50:59,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:50:59,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:50:59,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:50:59,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:50:59,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:50:59,309 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:50:59,309 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.865999937057. 2021-12-02 07:50:59,311 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:50:59,312 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:50:59,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:50:59,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:50:59,805 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:50:59,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:50:59,826 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:50:59,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:50:59,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:50:59,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:50:59,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:50:59,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:50:59,941 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:51:00,125 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:51:00,125 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.814999818802. 2021-12-02 07:51:00,127 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:51:00,128 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:51:00,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:51:00,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:51:00,546 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:51:00,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:51:00,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:51:00,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:51:00,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:51:00,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:51:00,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:51:00,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:51:00,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:51:00,862 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:51:00,864 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.736999988556. ############################################################ # # RAM FootPrint: 6006.87890625 # ############################################################ 2021-12-02 07:51:00,891 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 07:51:00,891 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 102 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 2021-12-02 07:52:42,895 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 07:52:42,895 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 07:52:47,898 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:52:48,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:52:49,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:52:49,881 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:52:49,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:52:49,895 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:52:49,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:52:49,928 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:52:49,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:52:49,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:52:49,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:52:50,009 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:52:50,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:52:50,101 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['15964 memcached\n'], []) 2021-12-02 07:52:50,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 07:52:50,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:52:50,226 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['15964 memcached\n'], []) 2021-12-02 07:52:50,227 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:52:50,230 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:52:50,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:52:50,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:52:50,737 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:52:50,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:52:50,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:52:50,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:52:50,818 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:52:50,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:52:50,862 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:52:50,882 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:52:50,898 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:52:50,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:52:50,971 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['98969 memcached\n'], []) 2021-12-02 07:52:50,971 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 07:52:51,048 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:52:51,112 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['98969 memcached\n'], []) 2021-12-02 07:52:51,114 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:52:51,118 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:52:51,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:52:51,525 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:52:51,618 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:52:51,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:52:51,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:52:51,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:52:51,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:52:51,719 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:52:51,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:52:51,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:52:51,779 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:52:51,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:52:51,848 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['223068 memcached\n'], []) 2021-12-02 07:52:51,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 07:52:51,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:52:51,946 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['223068 memcached\n'], []) 2021-12-02 07:52:51,947 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:52:51,948 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:52:52,084 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:52:52,410 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:52:52,516 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:52:52,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:52:52,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:52:52,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:52:52,612 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:52:52,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:52:52,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:52:52,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:52:52,697 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:52:52,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:52:52,789 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['4958 memcached\n'], []) 2021-12-02 07:52:52,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 07:52:52,872 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:52:52,914 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['4958 memcached\n'], []) 2021-12-02 07:52:52,914 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:52:52,917 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:52:53,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:52:53,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:52:53,362 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:52:53,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:52:53,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:52:53,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:52:53,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:52:53,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:52:53,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:52:53,477 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:52:53,494 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:52:53,510 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 07:52:53,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:52:53,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:53,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:53,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:52:53,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:52:53,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:53,775 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:53,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:52:53,862 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:52:53,895 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:53,941 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:53,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:52:54,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:52:54,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:54,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:54,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:52:54,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:52:54,186 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:54,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:54,232 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:52:54,247 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:52:54,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:52:54,276 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 07:52:54,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:52:54,288 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:54,303 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:52:54,315 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 07:52:54,315 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:52:54,319 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:52:54,446 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:52:56,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:52:56,246 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:52:56,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:52:56,270 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:52:56,290 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:52:56,313 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:52:56,336 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:52:56,357 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:52:56,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:52:56,400 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:52:56,418 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 07:52:56,446 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:52:56,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:56,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:56,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:52:56,576 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:52:56,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:56,647 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:56,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:52:56,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:52:56,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:56,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:56,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:52:56,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:52:56,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:52:56,815 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 07:52:56,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:52:56,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:56,846 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:52:56,861 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 07:52:56,861 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:52:56,865 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:52:56,982 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:52:57,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:52:57,329 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:52:57,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:52:57,357 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:52:57,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:52:57,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:52:57,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:52:57,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:52:57,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:52:57,479 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:52:57,494 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 07:52:57,522 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:52:57,538 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:57,571 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:57,617 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:52:57,650 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:52:57,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:57,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:57,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:52:57,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:52:57,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:57,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:57,910 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:52:57,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:52:57,954 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:57,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:58,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:52:58,048 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:52:58,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:58,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:58,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:52:58,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:52:58,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:58,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:58,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:52:58,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:52:58,341 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:58,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:58,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:52:58,431 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:52:58,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:58,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:58,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:52:58,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:52:58,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:52:58,584 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 07:52:58,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:52:58,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:58,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:52:58,638 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 07:52:58,638 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:52:58,641 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:52:58,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:52:59,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:52:59,157 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:52:59,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:52:59,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:52:59,191 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:52:59,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:52:59,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:52:59,244 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:52:59,257 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:52:59,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:52:59,286 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 07:52:59,323 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:52:59,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:59,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:59,418 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:52:59,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:52:59,474 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:59,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:59,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:52:59,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:52:59,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:59,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:59,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:52:59,673 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:52:59,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:59,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:59,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:52:59,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:52:59,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:59,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:52:59,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:52:59,877 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:52:59,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:52:59,907 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 07:52:59,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:52:59,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:52:59,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:52:59,950 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 07:52:59,950 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:52:59,950 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:52:59,953 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:53:00,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:53:00,352 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:53:00,421 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:53:00,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:53:00,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:53:00,456 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:53:00,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:53:00,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:53:00,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:53:00,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:53:00,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:00,648 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:53:00,654 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:00,654 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:02,655 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:02,815 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:53:02,822 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:02,822 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:04,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:05,000 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:53:05,005 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:05,006 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:07,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:07,180 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:53:07,188 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:07,188 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:09,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:09,374 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:53:09,380 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:09,380 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:11,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:11,546 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:53:11,551 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:11,551 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:13,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:13,724 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:53:13,730 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:13,730 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:15,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:15,903 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:53:15,907 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:15,907 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:17,908 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:18,059 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:18,059 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:20,061 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:20,234 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:20,234 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:22,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:22,382 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:22,382 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:24,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:24,535 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:24,535 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:26,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:26,696 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:26,696 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:28,696 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:28,838 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:28,838 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:30,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:30,979 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:30,979 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:32,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:33,127 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:33,128 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:35,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:35,342 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:35,342 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:37,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:37,568 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:37,568 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:39,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:39,796 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:39,796 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:41,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:41,979 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:41,979 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:43,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:44,187 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:44,187 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:46,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:46,388 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:46,388 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:48,390 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:48,559 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:48,559 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:50,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:50,763 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:50,763 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:52,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:52,934 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:52,934 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:54,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:55,161 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:55,163 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:57,163 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:57,375 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:57,377 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:53:59,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:53:59,585 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:53:59,585 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:01,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:01,733 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:01,733 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:03,734 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:03,953 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:03,954 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:05,954 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:06,112 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:06,112 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:08,112 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:08,275 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:08,276 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:10,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:10,437 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:10,437 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:12,438 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:12,598 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:12,599 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:14,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:14,759 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:14,759 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:16,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:16,927 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:16,927 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:18,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:19,091 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:19,091 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:21,091 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:21,253 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:21,253 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:23,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:23,434 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:23,434 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:25,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:25,588 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:25,588 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:27,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:27,746 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:27,746 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:29,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:29,914 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:29,914 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:31,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:32,069 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:32,069 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:34,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:34,233 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:34,233 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:36,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:36,391 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:36,392 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:38,392 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:38,558 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:38,559 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:40,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:40,730 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:40,730 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:42,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:42,901 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:42,901 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:44,903 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:45,075 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:45,075 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:47,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:47,240 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:47,240 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:49,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:49,418 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:54:49,420 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:54:51,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:51,588 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:54:51,588 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 111.638000011. 2021-12-02 07:54:51,589 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:54:51,591 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:54:51,734 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:54:54,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:54:54,844 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:54:54,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:54:54,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:54:54,887 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:54:54,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:54:54,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:54:55,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:54:55,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:54:55,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:55,263 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:54:55,263 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 3.67300009727. 2021-12-02 07:54:55,265 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:54:55,266 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:54:55,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:54:55,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:54:55,786 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:54:55,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:54:55,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:54:55,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:54:55,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:54:55,877 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:54:55,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:54:55,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:54:55,943 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:56,121 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:54:56,121 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.855999946594. 2021-12-02 07:54:56,121 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:54:56,124 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:54:56,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:54:56,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:54:56,559 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:54:56,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:54:56,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:54:56,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:54:56,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:54:56,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:54:56,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:54:56,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:54:56,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:54:56,951 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:54:56,951 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.830999851227. ############################################################ # # RAM FootPrint: 5976.54296875 # ############################################################ 2021-12-02 07:54:56,980 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 07:54:56,980 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 85 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 2021-12-02 07:56:21,984 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 07:56:21,984 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 07:56:26,986 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:56:27,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:56:27,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:56:27,608 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:56:27,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:56:27,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:56:27,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:56:27,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:56:27,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:56:27,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:56:27,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:56:27,901 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:56:27,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:56:27,983 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['17464 memcached\n'], []) 2021-12-02 07:56:27,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 07:56:28,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 07:56:28,112 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['17464 memcached\n'], []) 2021-12-02 07:56:28,112 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:56:28,115 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:56:28,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:56:28,647 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:56:28,759 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:56:28,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:56:28,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:56:28,818 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:56:28,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:56:28,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:56:28,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:56:28,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:56:28,953 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:56:28,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:56:29,023 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['99872 memcached\n'], []) 2021-12-02 07:56:29,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 07:56:29,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 07:56:29,233 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['99872 memcached\n'], []) 2021-12-02 07:56:29,234 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:56:29,236 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:56:29,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:56:29,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:56:29,742 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:56:29,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:56:29,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:56:29,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:56:29,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:56:29,841 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:56:29,859 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:56:29,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:56:29,901 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:56:29,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:56:29,977 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['223923 memcached\n'], []) 2021-12-02 07:56:29,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 07:56:30,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 07:56:30,078 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['223923 memcached\n'], []) 2021-12-02 07:56:30,078 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:56:30,082 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:56:30,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:56:30,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:56:30,578 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:56:30,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:56:30,602 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:56:30,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:56:30,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:56:30,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:56:30,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:56:30,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:56:30,721 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 07:56:30,721 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:56:30,776 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['6118 memcached\n'], []) 2021-12-02 07:56:30,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 07:56:30,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 07:56:30,901 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['6118 memcached\n'], []) 2021-12-02 07:56:30,901 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:56:30,904 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:56:31,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:56:31,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:56:31,358 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:56:31,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:56:31,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:56:31,392 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:56:31,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:56:31,433 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:56:31,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:56:31,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:56:31,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:56:31,496 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 07:56:31,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:56:31,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:31,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:31,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:56:31,670 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:56:31,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:31,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:31,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:56:31,823 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:56:31,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:31,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:31,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:56:31,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:56:31,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:32,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:32,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:56:32,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:56:32,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:32,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:32,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:56:32,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:56:32,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:56:32,161 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 07:56:32,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:56:32,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:32,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:56:32,198 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 07:56:32,198 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:56:32,200 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:56:32,301 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:56:32,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:56:32,678 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:56:32,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:56:32,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:56:32,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:56:32,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:56:32,755 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:56:32,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:56:32,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:56:32,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:56:32,821 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 07:56:32,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:56:32,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:32,908 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:32,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:56:32,984 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:56:33,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:33,040 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:33,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:56:33,091 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:56:33,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:33,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:33,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:56:33,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:56:33,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:56:33,203 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 07:56:33,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:56:33,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:33,234 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:56:33,250 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 07:56:33,250 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:56:33,253 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:56:33,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:56:33,619 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:56:33,701 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:56:33,701 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:56:33,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:56:33,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:56:33,765 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:56:33,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:56:33,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:56:33,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:56:33,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:56:33,859 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 07:56:33,885 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:56:33,910 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:33,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:34,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:56:34,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:56:34,084 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:34,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:34,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:56:34,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:56:34,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:34,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:34,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:56:34,395 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:56:34,426 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:34,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:34,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:56:34,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:56:34,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:34,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:34,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:56:34,619 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:56:34,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:34,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:34,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:56:34,701 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 07:56:34,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:34,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:34,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:56:34,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 07:56:34,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:34,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:34,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:56:34,849 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 07:56:34,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:56:34,878 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 07:56:34,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:56:34,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:34,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:56:34,917 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 07:56:34,917 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:56:34,921 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:56:35,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:56:35,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:56:35,362 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:56:35,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:56:35,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:56:35,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:56:35,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:56:35,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:56:35,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:56:35,446 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:56:35,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 07:56:35,476 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 07:56:35,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 07:56:35,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:35,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:35,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:56:35,607 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 07:56:35,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:35,653 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:35,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:56:35,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 07:56:35,752 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:35,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:35,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:56:35,862 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 07:56:35,897 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:35,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:35,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:56:36,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 07:56:36,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:36,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 07:56:36,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:56:36,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 07:56:36,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 07:56:36,111 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 07:56:36,112 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 07:56:36,127 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 07:56:36,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 07:56:36,157 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 07:56:36,157 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:56:36,157 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:56:36,161 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 07:56:36,265 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:56:37,714 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 07:56:37,785 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:56:37,785 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 07:56:37,801 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 07:56:37,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 07:56:37,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 07:56:37,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 07:56:37,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 07:56:37,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 07:56:37,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:56:38,003 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:56:38,012 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:56:38,012 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:56:40,012 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:56:40,167 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:56:40,171 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:56:40,171 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:56:42,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:56:42,308 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:56:42,315 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:56:42,315 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:56:44,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:56:44,476 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:56:44,482 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:56:44,482 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:56:46,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:56:46,654 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:56:46,658 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:56:46,658 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:56:48,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:56:48,842 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:56:48,846 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:56:48,848 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:56:50,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:56:51,017 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:56:51,023 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:56:51,023 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:56:53,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:56:53,188 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 07:56:53,194 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:56:53,194 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:56:55,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:56:55,339 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:56:55,339 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:56:57,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:56:57,486 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:56:57,486 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:56:59,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:56:59,657 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:56:59,657 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:01,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:01,818 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:01,818 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:03,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:03,969 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:03,969 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:05,969 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:06,114 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:06,115 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:08,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:08,259 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:08,259 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:10,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:10,433 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:10,433 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:12,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:12,617 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:12,617 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:14,617 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:14,822 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:14,822 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:16,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:17,055 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:17,055 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:19,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:19,289 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:19,289 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:21,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:21,486 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:21,486 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:23,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:23,680 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:23,680 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:25,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:25,855 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:25,855 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:27,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:28,069 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:28,069 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:30,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:30,290 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:30,290 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:32,290 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:32,513 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:32,513 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:34,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:34,714 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:34,714 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:36,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:36,914 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:36,915 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:38,915 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:39,117 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:39,117 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:41,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:41,273 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:41,273 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:43,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:43,434 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:43,434 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:45,436 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:45,598 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:45,598 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:47,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:47,759 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:47,759 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:49,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:49,924 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:49,924 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:51,926 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:52,088 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:52,088 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:54,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:54,265 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:54,265 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:56,265 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:56,440 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:56,440 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:57:58,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:57:58,608 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:57:58,609 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:00,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:00,766 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:00,766 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:02,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:02,946 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:02,947 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:04,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:05,128 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:05,130 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:07,130 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:07,293 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:07,293 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:09,293 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:09,451 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:09,451 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:11,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:11,644 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:11,644 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:13,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:13,828 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:13,828 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:15,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:16,006 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:16,006 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:18,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:18,187 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:18,187 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:20,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:20,371 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:20,371 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:22,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:22,549 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:22,549 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:24,551 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:24,759 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:24,760 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:26,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:26,937 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 07:58:26,937 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 07:58:28,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:29,125 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 07:58:29,127 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 112.969000101. 2021-12-02 07:58:29,128 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:58:29,130 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 07:58:29,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:58:29,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 07:58:29,654 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:58:29,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 07:58:29,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 07:58:29,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 07:58:29,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 07:58:29,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 07:58:29,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 07:58:29,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 07:58:29,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:30,048 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 07:58:30,048 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.920000076294. 2021-12-02 07:58:30,049 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:58:30,052 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 07:58:30,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:58:30,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 07:58:30,579 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:58:30,579 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 07:58:30,604 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 07:58:30,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 07:58:30,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 07:58:30,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 07:58:30,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 07:58:30,719 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 07:58:30,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:30,930 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 07:58:30,930 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.881000041962. 2021-12-02 07:58:30,931 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 07:58:30,934 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 07:58:31,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 07:58:31,311 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 07:58:31,398 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 07:58:31,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 07:58:31,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 07:58:31,443 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 07:58:31,471 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 07:58:31,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 07:58:31,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 07:58:31,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 07:58:31,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 07:58:31,772 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 07:58:31,772 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.841000080109. ############################################################ # # RAM FootPrint: 5963.0859375 # ############################################################ 2021-12-02 07:58:31,812 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 07:58:31,812 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 110 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 2021-12-02 08:00:21,819 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 08:00:21,819 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 08:00:26,822 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:00:26,982 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:00:27,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:00:27,461 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:00:27,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:00:27,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:00:27,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:00:27,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:00:27,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:00:27,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:00:27,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:00:27,661 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:00:27,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:00:27,733 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['18972 memcached\n'], []) 2021-12-02 08:00:27,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 08:00:27,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:00:27,903 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['18972 memcached\n'], []) 2021-12-02 08:00:27,903 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:00:27,905 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:00:28,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:00:28,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:00:28,440 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:00:28,441 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:00:28,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:00:28,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:00:28,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:00:28,502 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:00:28,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:00:28,619 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:00:28,638 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:00:28,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:00:28,717 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['100756 memcached\n'], []) 2021-12-02 08:00:28,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 08:00:28,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:00:28,901 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['100756 memcached\n'], []) 2021-12-02 08:00:28,901 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:00:28,905 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:00:29,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:00:29,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:00:29,387 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:00:29,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:00:29,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:00:29,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:00:29,456 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:00:29,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:00:29,499 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:00:29,517 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:00:29,538 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:00:29,538 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:00:29,599 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['224768 memcached\n'], []) 2021-12-02 08:00:29,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 08:00:29,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:00:29,740 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['224768 memcached\n'], []) 2021-12-02 08:00:29,740 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:00:29,743 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:00:29,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:00:30,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:00:30,290 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:00:30,290 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:00:30,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:00:30,392 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:00:30,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:00:30,446 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:00:30,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:00:30,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:00:30,506 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:00:30,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:00:30,589 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['7255 memcached\n'], []) 2021-12-02 08:00:30,591 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 08:00:30,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:00:30,773 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['7255 memcached\n'], []) 2021-12-02 08:00:30,773 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:00:30,776 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:00:30,903 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:00:31,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:00:31,255 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:00:31,255 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:00:31,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:00:31,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:00:31,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:00:31,329 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:00:31,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:00:31,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:00:31,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:00:31,388 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 08:00:31,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:00:31,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:31,482 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:31,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:00:31,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:00:31,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:31,647 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:31,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:00:31,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:00:31,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:31,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:31,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:00:31,897 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:00:31,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:31,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:32,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:00:32,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:00:32,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:32,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:32,114 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:00:32,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:00:32,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:00:32,167 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 08:00:32,168 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:00:32,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:32,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:00:32,210 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 08:00:32,210 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:00:32,213 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:00:32,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:00:32,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:00:32,637 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:00:32,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:00:32,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:00:32,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:00:32,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:00:32,701 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:00:32,714 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:00:32,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:00:32,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:00:32,759 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 08:00:32,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:00:32,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:32,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:32,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:00:32,931 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:00:32,959 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:32,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:33,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:00:33,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:00:33,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:33,099 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:33,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:00:33,153 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:00:33,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:00:33,191 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 08:00:33,191 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:00:33,209 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:33,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:00:33,243 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 08:00:33,243 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:00:33,246 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:00:33,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:00:33,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:00:33,723 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:00:33,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:00:33,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:00:33,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:00:33,783 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:00:33,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:00:33,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:00:33,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:00:33,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:00:33,871 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 08:00:33,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:00:33,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:33,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:34,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:00:34,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:00:34,068 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:34,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:34,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:00:34,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:00:34,219 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:34,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:34,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:00:34,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:00:34,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:34,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:34,443 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:00:34,471 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:00:34,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:34,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:34,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:00:34,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:00:34,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:34,673 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:34,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:00:34,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:00:34,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:34,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:34,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:00:34,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:00:34,943 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:34,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:35,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:00:35,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:00:35,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:00:35,125 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 08:00:35,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:00:35,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:35,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:00:35,193 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 08:00:35,193 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:00:35,197 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:00:35,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:00:35,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:00:35,676 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:00:35,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:00:35,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:00:35,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:00:35,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:00:35,734 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:00:35,747 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:00:35,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:00:35,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:00:35,786 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 08:00:35,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:00:35,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:35,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:35,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:00:35,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:00:35,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:36,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:36,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:00:36,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:00:36,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:36,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:36,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:00:36,270 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:00:36,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:36,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:36,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:00:36,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:00:36,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:36,448 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:00:36,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:00:36,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:00:36,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:00:36,515 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 08:00:36,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:00:36,528 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:00:36,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:00:36,555 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 08:00:36,555 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:00:36,555 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:00:36,569 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:00:36,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:00:36,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:00:37,032 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:00:37,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:00:37,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:00:37,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:00:37,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:00:37,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:00:37,122 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:00:37,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:00:37,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:00:37,299 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: 'ep_warmup_thread' 2021-12-02 08:00:37,305 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:00:37,305 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:00:39,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:00:39,414 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:00:39,420 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:00:39,420 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:00:41,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:00:41,607 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:00:41,611 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:00:41,611 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:00:43,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:00:43,793 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:00:43,801 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:00:43,802 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:00:45,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:00:45,970 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:00:45,974 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:00:45,976 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:00:47,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:00:48,164 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:00:48,170 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:00:48,170 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:00:50,170 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:00:50,358 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:00:50,364 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:00:50,364 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:00:52,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:00:52,529 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:00:52,535 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:00:52,535 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:00:54,535 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:00:54,684 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:00:54,684 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:00:56,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:00:56,832 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:00:56,832 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:00:58,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:00:58,980 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:00:58,980 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:00,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:01,131 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:01,131 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:03,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:03,290 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:03,290 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:05,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:05,440 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:05,440 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:07,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:07,586 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:07,586 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:09,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:09,766 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:09,766 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:11,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:11,969 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:11,969 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:13,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:14,198 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:14,198 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:16,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:16,424 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:16,424 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:18,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:18,631 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:18,631 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:20,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:20,826 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:20,826 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:22,826 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:22,983 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:22,983 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:24,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:25,151 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:25,151 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:27,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:27,380 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:27,381 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:29,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:29,586 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:29,586 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:31,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:31,792 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:31,792 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:33,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:33,994 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:33,996 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:35,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:36,200 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:36,201 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:38,201 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:38,415 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:38,415 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:40,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:40,568 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:40,569 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:42,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:42,732 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:42,732 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:44,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:44,914 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:44,914 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:46,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:47,069 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:47,071 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:49,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:49,234 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:49,234 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:51,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:51,390 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:51,390 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:53,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:53,566 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:53,566 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:55,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:55,743 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:55,743 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:57,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:01:57,930 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:01:57,930 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:01:59,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:00,121 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:02:00,121 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:02:02,122 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:02,298 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:02:02,298 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:02:04,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:04,483 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:02:04,484 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:02:06,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:06,657 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:02:06,657 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:02:08,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:08,878 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:02:08,878 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:02:10,880 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:11,055 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:02:11,056 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:02:13,056 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:13,223 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:02:13,223 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:02:15,224 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:15,407 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:02:15,407 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:02:17,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:17,572 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:02:17,572 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:02:19,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:19,736 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:02:19,736 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:02:21,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:21,910 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:02:21,910 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:02:23,910 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:24,065 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:02:24,065 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:02:26,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:26,229 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:02:26,230 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 109.674000025. 2021-12-02 08:02:26,230 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:02:26,233 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:02:26,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:02:32,568 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:02:32,644 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:02:32,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:02:32,709 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:02:32,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:02:32,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:02:32,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:02:32,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:02:32,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:02:32,862 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:33,030 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:02:33,032 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 6.79999995232. 2021-12-02 08:02:33,032 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:02:33,035 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:02:33,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:02:33,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:02:33,569 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:02:33,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:02:33,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:02:33,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:02:33,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:02:33,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:02:33,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:02:33,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:02:33,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:33,953 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:02:33,953 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.921999931335. 2021-12-02 08:02:33,954 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:02:33,957 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:02:34,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:02:34,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:02:34,523 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:02:34,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:02:34,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:02:34,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:02:34,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:02:34,640 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:02:34,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:02:34,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:02:34,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:02:34,936 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:02:34,937 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.982000112534. ############################################################ # # RAM FootPrint: 5950.0859375 # ############################################################ 2021-12-02 08:02:34,973 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 08:02:34,973 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 98 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 2021-12-02 08:04:12,979 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 08:04:12,979 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 08:04:17,982 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:04:18,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:04:18,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:04:18,562 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:04:18,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:04:18,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:04:18,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:04:18,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:04:18,742 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:04:18,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:04:18,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:04:18,793 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:04:18,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:04:18,865 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['20577 memcached\n'], []) 2021-12-02 08:04:18,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 08:04:19,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:04:19,082 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['20577 memcached\n'], []) 2021-12-02 08:04:19,082 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:04:19,085 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:04:19,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:04:19,502 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:04:19,566 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:04:19,568 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:04:19,640 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:04:19,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:04:19,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:04:19,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:04:19,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:04:19,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:04:19,769 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:04:19,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:04:19,871 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['101671 memcached\n'], []) 2021-12-02 08:04:19,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 08:04:20,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:04:20,088 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['101671 memcached\n'], []) 2021-12-02 08:04:20,088 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:04:20,092 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:04:20,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:04:20,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:04:20,601 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:04:20,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:04:20,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:04:20,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:04:20,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:04:20,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:04:20,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:04:20,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:04:20,746 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:04:20,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:04:20,822 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['225620 memcached\n'], []) 2021-12-02 08:04:20,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 08:04:20,882 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:04:20,953 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['225620 memcached\n'], []) 2021-12-02 08:04:20,953 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:04:20,957 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:04:21,130 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:04:21,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:04:21,493 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:04:21,494 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:04:21,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:04:21,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:04:21,573 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:04:21,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:04:21,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:04:21,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:04:21,710 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:04:21,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:04:21,792 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['8396 memcached\n'], []) 2021-12-02 08:04:21,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 08:04:21,859 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:04:21,960 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['8396 memcached\n'], []) 2021-12-02 08:04:21,960 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:04:21,963 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:04:22,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:04:22,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:04:22,427 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:04:22,428 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:04:22,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:04:22,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:04:22,477 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:04:22,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:04:22,507 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:04:22,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:04:22,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:04:22,546 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 08:04:22,573 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:04:22,622 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:22,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:22,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:04:22,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:04:22,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:22,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:22,821 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:04:22,846 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:04:22,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:22,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:22,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:04:22,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:04:22,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:22,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:23,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:04:23,048 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:04:23,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:23,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:23,140 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:04:23,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:04:23,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:04:23,206 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 08:04:23,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:04:23,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:23,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:04:23,259 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 08:04:23,259 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:04:23,263 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:04:23,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:04:24,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:04:24,144 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:04:24,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:04:24,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:04:24,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:04:24,201 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:04:24,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:04:24,237 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:04:24,252 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:04:24,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:04:24,282 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 08:04:24,311 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:04:24,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:24,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:24,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:04:24,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:04:24,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:24,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:24,563 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:04:24,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:04:24,619 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:24,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:24,686 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:04:24,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:04:24,734 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:04:24,753 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 08:04:24,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:04:24,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:24,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:04:24,805 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 08:04:24,805 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:04:24,809 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:04:24,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:04:25,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:04:25,253 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:04:25,255 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:04:25,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:04:25,290 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:04:25,311 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:04:25,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:04:25,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:04:25,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:04:25,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:04:25,391 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 08:04:25,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:04:25,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:25,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:25,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:04:25,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:04:25,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:25,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:25,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:04:25,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:04:25,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:25,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:25,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:04:25,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:04:25,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:25,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:25,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:04:26,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:04:26,053 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:26,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:26,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:04:26,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:04:26,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:26,247 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:26,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:04:26,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:04:26,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:26,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:26,418 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:04:26,446 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:04:26,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:26,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:26,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:04:26,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:04:26,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:26,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:26,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:04:26,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:04:26,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:04:26,684 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 08:04:26,686 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:04:26,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:26,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:04:26,733 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 08:04:26,733 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:04:26,736 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:04:26,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:04:27,122 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:04:27,200 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:04:27,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:04:27,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:04:27,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:04:27,260 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:04:27,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:04:27,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:04:27,311 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:04:27,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:04:27,354 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 08:04:27,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:04:27,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:27,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:27,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:04:27,535 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:04:27,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:27,612 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:27,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:04:27,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:04:27,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:27,775 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:27,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:04:27,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:04:27,887 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:27,926 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:27,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:04:27,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:04:28,022 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:28,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:04:28,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:04:28,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:04:28,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:04:28,134 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 08:04:28,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:04:28,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:04:28,170 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:04:28,184 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 08:04:28,184 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:04:28,184 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:04:28,187 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:04:28,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:04:28,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:04:28,654 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:04:28,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:04:28,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:04:28,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:04:28,719 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:04:28,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:04:28,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:04:28,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:04:28,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:28,964 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:04:28,970 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:28,970 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:30,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:31,153 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:04:31,157 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:31,157 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:33,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:33,322 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:04:33,328 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:33,328 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:35,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:35,482 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:04:35,486 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:35,486 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:37,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:37,657 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:04:37,663 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:37,663 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:39,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:39,782 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:04:39,786 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:39,786 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:41,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:41,953 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:04:41,963 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:41,963 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:43,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:44,109 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:44,111 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:46,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:46,309 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:46,311 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:48,311 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:48,463 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:48,463 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:50,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:50,609 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:50,609 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:52,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:52,756 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:52,757 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:54,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:54,901 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:54,901 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:56,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:57,045 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:57,045 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:04:59,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:04:59,201 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:04:59,201 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:01,201 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:01,430 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:01,430 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:03,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:03,638 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:03,638 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:05,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:05,780 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:05,780 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:07,780 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:07,983 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:07,983 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:09,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:10,181 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:10,181 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:12,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:12,408 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:12,408 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:14,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:14,618 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:14,619 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:16,619 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:16,838 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:16,838 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:18,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:19,013 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:19,013 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:21,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:21,220 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:21,220 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:23,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:23,444 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:23,444 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:25,446 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:25,625 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:25,625 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:27,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:27,832 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:27,834 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:29,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:30,026 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:30,026 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:32,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:32,186 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:32,186 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:34,186 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:34,348 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:34,348 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:36,349 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:36,509 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:36,509 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:38,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:38,665 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:38,665 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:40,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:40,829 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:40,829 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:42,831 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:42,996 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:42,997 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:44,997 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:45,165 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:45,167 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:47,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:47,331 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:47,331 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:49,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:49,490 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:49,490 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:51,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:51,648 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:51,650 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:53,650 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:53,849 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:53,849 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:55,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:56,035 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:56,035 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:05:58,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:05:58,221 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:05:58,221 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:06:00,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:00,397 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:06:00,397 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:06:02,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:02,568 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:06:02,568 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:06:04,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:04,753 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:06:04,753 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:06:06,755 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:06,923 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:06:06,923 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:06:08,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:09,101 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:06:09,101 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:06:11,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:11,265 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:06:11,265 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:06:13,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:13,440 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:06:13,440 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:06:15,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:15,608 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:06:15,608 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:06:17,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:17,786 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:06:17,786 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:06:19,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:23,285 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:06:23,286 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 115.101000071. 2021-12-02 08:06:23,286 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:06:23,289 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:06:23,418 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:06:23,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:06:23,921 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:06:23,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:06:23,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:06:23,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:06:24,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:06:24,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:06:24,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:06:24,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:06:24,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:24,391 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:06:24,392 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.10599994659. 2021-12-02 08:06:24,394 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:06:24,395 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:06:24,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:06:24,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:06:24,923 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:06:24,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:06:24,946 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:06:24,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:06:24,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:06:25,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:06:25,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:06:25,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:06:25,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:25,286 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:06:25,288 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.894000053406. 2021-12-02 08:06:25,289 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:06:25,289 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:06:25,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:06:25,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:06:25,762 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:06:25,762 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:06:25,785 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:06:25,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:06:25,826 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:06:25,849 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:06:25,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:06:25,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:06:25,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:06:26,065 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:06:26,065 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.776999950409. ############################################################ # # RAM FootPrint: 5923.06640625 # ############################################################ 2021-12-02 08:06:26,098 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 08:06:26,098 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 105 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 2021-12-02 08:08:11,102 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 08:08:11,102 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 08:08:16,105 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:08:16,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:08:16,551 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:08:16,628 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:08:16,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:08:16,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:08:16,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:08:16,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:08:16,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:08:17,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:08:17,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:08:17,046 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:08:17,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:08:17,128 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['22018 memcached\n'], []) 2021-12-02 08:08:17,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 08:08:17,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:08:17,365 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['22018 memcached\n'], []) 2021-12-02 08:08:17,365 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:08:17,368 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:08:17,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:08:17,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:08:17,786 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:08:17,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:08:17,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:08:17,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:08:17,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:08:17,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:08:17,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:08:17,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:08:17,917 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:08:17,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:08:17,963 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['102576 memcached\n'], []) 2021-12-02 08:08:17,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 08:08:18,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:08:18,096 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['102576 memcached\n'], []) 2021-12-02 08:08:18,096 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:08:18,101 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:08:18,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:08:18,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:08:18,589 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:08:18,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:08:18,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:08:18,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:08:18,655 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:08:18,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:08:18,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:08:18,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:08:18,739 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:08:18,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:08:18,812 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['226491 memcached\n'], []) 2021-12-02 08:08:18,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 08:08:18,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:08:18,934 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['226491 memcached\n'], []) 2021-12-02 08:08:18,934 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:08:18,937 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:08:19,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:08:19,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:08:19,440 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:08:19,441 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:08:19,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:08:19,556 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:08:19,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:08:19,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:08:19,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:08:19,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:08:19,687 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:08:19,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:08:19,772 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['9601 memcached\n'], []) 2021-12-02 08:08:19,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 08:08:19,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:08:19,976 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['9601 memcached\n'], []) 2021-12-02 08:08:19,976 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:08:19,980 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:08:20,096 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:08:20,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:08:20,440 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:08:20,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:08:20,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:08:20,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:08:20,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:08:20,522 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:08:20,538 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:08:20,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:08:20,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:08:20,585 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 08:08:20,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:08:20,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:20,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:20,719 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:08:20,755 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:08:20,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:20,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:20,872 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:08:20,908 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:08:20,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:20,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:21,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:08:21,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:08:21,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:21,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:21,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:08:21,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:08:21,237 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:21,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:21,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:08:21,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:08:21,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:08:21,364 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 08:08:21,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:08:21,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:21,395 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:08:21,411 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 08:08:21,411 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:08:21,413 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:08:21,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:08:21,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:08:21,897 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:08:21,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:08:21,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:08:21,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:08:21,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:08:21,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:08:21,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:08:22,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:08:22,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:08:22,040 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 08:08:22,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:08:22,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:22,132 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:22,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:08:22,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:08:22,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:22,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:22,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:08:22,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:08:22,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:22,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:22,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:08:22,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:08:22,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:08:22,460 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 08:08:22,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:08:22,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:22,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:08:22,505 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 08:08:22,505 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:08:22,509 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:08:22,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:08:22,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:08:22,973 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:08:22,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:08:22,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:08:23,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:08:23,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:08:23,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:08:23,076 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:08:23,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:08:23,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:08:23,130 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 08:08:23,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:08:23,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:23,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:23,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:08:23,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:08:23,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:23,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:23,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:08:23,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:08:23,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:23,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:23,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:08:23,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:08:23,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:23,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:23,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:08:23,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:08:23,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:23,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:23,903 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:08:23,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:08:23,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:23,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:24,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:08:24,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:08:24,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:24,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:24,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:08:24,183 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:08:24,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:24,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:24,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:08:24,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:08:24,367 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:24,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:24,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:08:24,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:08:24,474 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:08:24,493 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 08:08:24,494 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:08:24,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:24,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:08:24,551 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 08:08:24,551 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:08:24,553 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:08:24,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:08:24,941 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:08:25,019 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:08:25,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:08:25,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:08:25,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:08:25,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:08:25,076 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:08:25,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:08:25,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:08:25,114 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:08:25,127 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 08:08:25,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:08:25,214 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:25,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:25,311 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:08:25,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:08:25,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:25,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:25,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:08:25,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:08:25,507 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:25,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:25,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:08:25,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:08:25,604 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:25,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:25,658 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:08:25,678 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:08:25,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:25,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:08:25,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:08:25,762 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:08:25,778 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:08:25,789 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 08:08:25,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:08:25,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:08:25,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:08:25,826 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 08:08:25,828 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:08:25,828 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:08:25,832 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:08:25,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:08:26,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:08:26,316 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:08:26,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:08:26,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:08:26,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:08:26,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:08:26,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:08:26,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:08:26,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:08:26,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:26,586 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-02 08:08:26,592 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:26,592 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:28,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:28,762 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:08:28,767 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:28,767 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:30,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:30,948 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:08:30,954 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:30,954 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:32,954 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:33,132 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:08:33,137 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:33,137 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:35,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:35,301 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:08:35,305 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:35,305 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:37,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:37,480 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:08:37,483 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:37,483 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:39,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:39,667 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:08:39,671 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:39,671 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:41,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:41,845 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:08:41,851 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:41,851 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:43,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:44,013 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:08:44,019 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:44,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:46,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:46,167 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:46,168 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:48,168 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:48,312 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:48,312 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:50,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:50,464 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:50,464 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:52,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:52,614 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:52,614 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:54,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:54,759 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:54,759 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:56,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:56,914 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:56,914 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:08:58,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:08:59,055 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:08:59,055 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:01,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:01,234 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:01,236 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:03,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:03,457 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:03,457 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:05,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:05,687 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:05,687 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:07,688 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:07,884 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:07,885 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:09,885 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:10,115 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:10,115 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:12,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:12,348 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:12,348 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:14,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:14,572 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:14,572 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:16,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:16,789 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:16,789 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:18,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:18,973 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:18,973 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:20,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:21,128 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:21,128 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:23,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:23,319 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:23,319 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:25,319 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:25,520 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:25,522 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:27,522 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:27,726 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:27,726 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:29,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:29,967 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:29,967 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:31,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:32,128 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:32,128 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:34,130 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:34,295 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:34,295 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:36,295 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:36,459 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:36,460 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:38,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:38,611 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:38,611 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:40,612 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:40,766 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:40,766 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:42,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:42,920 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:42,920 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:44,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:45,092 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:45,092 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:47,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:47,256 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:47,256 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:49,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:49,418 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:49,418 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:51,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:51,595 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:51,595 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:53,596 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:53,775 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:53,776 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:55,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:55,983 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:55,983 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:09:57,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:09:58,151 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:09:58,151 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:10:00,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:00,328 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:10:00,328 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:10:02,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:02,494 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:10:02,494 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:10:04,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:04,661 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:10:04,661 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:10:06,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:06,825 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:10:06,826 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:10:08,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:08,992 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:10:08,992 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:10:10,992 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:11,155 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:10:11,155 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:10:13,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:13,316 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:10:13,316 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:10:15,318 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:15,506 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:10:15,506 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:10:17,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:17,674 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:10:17,676 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:10:19,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:19,901 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:10:19,903 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 114.073999882. 2021-12-02 08:10:19,903 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:10:19,905 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:10:20,617 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:10:21,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:10:21,908 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:10:21,910 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:10:21,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:10:21,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:10:22,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:10:22,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:10:22,053 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:10:22,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:10:22,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:22,253 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:10:22,253 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 2.34999990463. 2021-12-02 08:10:22,253 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:10:22,256 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:10:22,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:10:22,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:10:22,747 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:10:22,747 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:10:22,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:10:22,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:10:22,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:10:22,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:10:22,854 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:10:22,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:10:22,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:23,099 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:10:23,099 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.845999956131. 2021-12-02 08:10:23,099 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:10:23,102 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:10:23,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:10:23,499 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:10:23,586 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:10:23,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:10:23,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:10:23,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:10:23,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:10:23,665 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:10:23,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:10:23,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:10:23,826 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:10:23,996 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:10:23,996 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.896000146866. ############################################################ # # RAM FootPrint: 5876.6640625 # ############################################################ 2021-12-02 08:10:24,028 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 08:10:24,028 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 88 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 2021-12-02 08:11:52,032 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 08:11:52,032 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 08:11:57,035 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:11:57,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:11:58,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:11:58,969 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:11:58,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:11:58,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:11:59,005 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:11:59,022 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:11:59,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:11:59,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:11:59,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:11:59,082 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:11:59,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:11:59,155 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['23602 memcached\n'], []) 2021-12-02 08:11:59,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 08:11:59,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:11:59,278 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['23602 memcached\n'], []) 2021-12-02 08:11:59,278 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:11:59,280 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:11:59,438 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:11:59,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:11:59,904 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:11:59,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:11:59,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:11:59,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:12:00,022 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:12:00,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:12:00,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:12:00,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:12:00,137 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:12:00,137 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:12:00,252 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['103484 memcached\n'], []) 2021-12-02 08:12:00,252 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 08:12:00,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:12:00,540 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['103484 memcached\n'], []) 2021-12-02 08:12:00,540 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:12:00,543 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:12:00,670 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:12:00,959 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:12:01,043 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:12:01,045 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:12:01,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:12:01,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:12:01,112 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:12:01,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:12:01,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:12:01,178 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:12:01,203 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:12:01,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:12:01,292 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['227344 memcached\n'], []) 2021-12-02 08:12:01,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 08:12:01,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:12:01,431 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['227344 memcached\n'], []) 2021-12-02 08:12:01,431 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:12:01,434 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:12:01,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:12:01,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:12:01,917 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:12:01,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:12:01,946 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:12:01,982 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:12:02,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:12:02,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:12:02,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:12:02,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:12:02,098 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:12:02,099 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:12:02,249 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['10790 memcached\n'], []) 2021-12-02 08:12:02,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 08:12:02,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:12:02,431 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['10790 memcached\n'], []) 2021-12-02 08:12:02,431 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:12:02,434 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:12:02,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:12:02,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:12:02,901 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:12:02,903 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:12:02,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:12:02,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:12:02,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:12:02,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:12:02,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:12:03,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:12:03,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:12:03,038 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 08:12:03,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:12:03,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:03,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:03,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:12:03,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:12:03,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:03,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:03,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:12:03,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:12:03,301 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:03,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:03,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:12:03,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:12:03,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:03,423 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:03,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:12:03,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:12:03,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:03,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:03,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:12:03,548 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:12:03,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:12:03,575 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 08:12:03,576 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:12:03,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:03,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:12:03,612 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 08:12:03,612 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:12:03,615 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:12:03,737 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:12:04,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:12:04,108 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:12:04,109 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:12:04,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:12:04,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:12:04,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:12:04,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:12:04,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:12:04,227 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:12:04,244 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:12:04,259 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 08:12:04,290 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:12:04,313 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:04,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:04,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:12:04,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:12:04,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:04,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:04,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:12:04,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:12:04,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:04,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:04,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:12:04,640 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:12:04,658 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:12:04,671 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 08:12:04,673 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:12:04,686 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:04,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:12:04,719 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 08:12:04,719 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:12:04,720 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:12:04,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:12:05,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:12:05,178 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:12:05,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:12:05,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:12:05,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:12:05,232 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:12:05,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:12:05,262 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:12:05,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:12:05,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:12:05,306 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 08:12:05,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:12:05,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:05,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:05,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:12:05,436 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:12:05,456 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:05,482 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:05,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:12:05,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:12:05,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:05,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:05,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:12:05,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:12:05,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:05,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:05,785 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:12:05,806 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:12:05,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:05,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:05,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:12:05,897 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:12:05,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:05,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:05,954 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:12:05,971 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:12:05,987 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:06,010 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:06,030 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:12:06,048 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:12:06,063 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:06,084 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:06,104 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:12:06,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:12:06,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:06,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:06,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:12:06,186 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:12:06,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:12:06,213 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 08:12:06,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:12:06,224 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:06,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:12:06,289 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 08:12:06,289 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:12:06,292 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:12:06,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:12:06,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:12:06,750 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:12:06,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:12:06,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:12:06,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:12:06,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:12:06,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:12:06,841 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:12:06,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:12:06,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:12:06,887 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 08:12:06,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:12:06,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:06,982 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:07,030 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:12:07,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:12:07,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:07,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:07,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:12:07,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:12:07,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:07,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:07,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:12:07,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:12:07,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:07,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:07,492 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:12:07,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:12:07,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:07,576 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:12:07,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:12:07,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:12:07,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:12:07,661 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 08:12:07,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:12:07,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:12:07,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:12:07,709 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 08:12:07,709 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:12:07,709 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:12:07,711 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:12:07,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:12:08,096 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:12:08,177 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:12:08,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:12:08,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:12:08,216 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:12:08,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:12:08,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:12:08,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:12:08,288 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:12:08,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:08,459 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: 'ep_warmup_thread' 2021-12-02 08:12:08,464 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:08,464 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:10,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:10,634 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:12:10,641 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:10,641 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:12,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:12,812 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:12:12,819 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:12,819 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:14,821 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:14,993 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:12:14,999 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:14,999 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:16,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:17,144 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:12:17,150 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:17,150 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:19,150 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:19,332 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:12:19,336 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:19,336 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:21,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:21,516 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:12:21,520 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:21,520 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:23,520 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:23,697 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:12:23,701 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:23,703 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:25,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:25,842 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:25,842 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:27,844 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:27,990 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:27,990 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:29,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:30,150 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:30,150 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:32,150 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:32,308 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:32,308 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:34,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:34,470 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:34,470 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:36,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:36,611 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:36,611 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:38,612 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:38,805 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:38,805 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:40,806 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:41,013 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:41,013 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:43,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:43,246 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:43,247 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:45,247 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:45,486 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:45,486 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:47,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:47,665 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:47,665 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:49,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:49,892 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:49,892 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:51,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:52,125 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:52,125 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:54,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:54,331 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:54,331 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:56,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:56,522 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:56,522 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:12:58,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:12:58,739 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:12:58,739 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:00,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:00,959 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:00,960 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:02,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:03,171 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:03,171 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:05,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:05,374 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:05,374 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:07,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:07,578 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:07,578 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:09,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:09,898 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:09,898 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:11,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:12,061 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:12,061 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:14,061 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:14,220 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:14,220 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:16,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:16,371 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:16,372 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:18,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:18,532 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:18,532 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:20,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:20,694 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:20,694 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:22,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:22,858 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:22,858 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:24,859 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:25,028 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:25,029 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:27,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:27,203 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:27,203 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:29,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:29,359 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:29,361 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:31,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:31,523 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:31,523 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:33,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:33,690 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:33,691 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:35,691 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:35,859 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:35,861 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:37,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:38,032 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:38,032 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:40,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:40,197 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:40,197 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:42,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:42,372 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:42,372 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:44,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:44,526 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:44,526 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:46,528 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:46,711 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:46,711 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:48,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:48,891 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:48,891 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:50,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:51,069 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:51,071 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:53,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:53,247 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:53,249 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:55,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:55,426 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:13:55,427 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:13:57,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:57,559 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:13:57,559 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 109.849999905. 2021-12-02 08:13:57,559 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:13:57,562 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:13:57,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:13:57,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:13:58,003 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:13:58,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:13:58,051 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:13:58,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:13:58,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:13:58,130 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:13:58,191 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:13:58,206 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:13:58,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:58,391 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:13:58,391 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.832999944687. 2021-12-02 08:13:58,391 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:13:58,394 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:13:58,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:13:58,793 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:13:58,880 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:13:58,880 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:13:58,900 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:13:58,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:13:58,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:13:58,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:13:58,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:13:59,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:13:59,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:59,207 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:13:59,207 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.815999984741. 2021-12-02 08:13:59,209 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:13:59,210 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:13:59,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:13:59,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:13:59,622 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:13:59,622 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:13:59,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:13:59,655 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:13:59,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:13:59,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:13:59,742 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:13:59,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:13:59,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:13:59,911 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:13:59,911 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.702999830246. ############################################################ # # RAM FootPrint: 5877.9375 # ############################################################ 2021-12-02 08:13:59,943 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 08:13:59,943 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 105 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 2021-12-02 08:15:44,947 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 08:15:44,948 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 08:15:49,951 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:15:50,114 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:15:50,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:15:50,502 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:15:50,502 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:15:50,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:15:50,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:15:50,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:15:50,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:15:50,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:15:50,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:15:50,642 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:15:50,642 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:15:50,693 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['25128 memcached\n'], []) 2021-12-02 08:15:50,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 08:15:50,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:15:50,811 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['25128 memcached\n'], []) 2021-12-02 08:15:50,812 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:15:50,815 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:15:50,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:15:51,319 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:15:51,405 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:15:51,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:15:51,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:15:51,553 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:15:51,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:15:51,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:15:51,647 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:15:51,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:15:51,711 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:15:51,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:15:51,773 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['104368 memcached\n'], []) 2021-12-02 08:15:51,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 08:15:51,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:15:51,907 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['104368 memcached\n'], []) 2021-12-02 08:15:51,907 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:15:51,911 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:15:52,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:15:52,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:15:52,408 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:15:52,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:15:52,428 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:15:52,448 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:15:52,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:15:52,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:15:52,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:15:52,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:15:52,555 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:15:52,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:15:52,628 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['228191 memcached\n'], []) 2021-12-02 08:15:52,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 08:15:52,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:15:52,752 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['228191 memcached\n'], []) 2021-12-02 08:15:52,752 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:15:52,755 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:15:52,887 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:15:53,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:15:53,364 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:15:53,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:15:53,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:15:53,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:15:53,441 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:15:53,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:15:53,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:15:53,502 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:15:53,526 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:15:53,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:15:53,732 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['11968 memcached\n'], []) 2021-12-02 08:15:53,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 08:15:53,793 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:15:53,832 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['11968 memcached\n'], []) 2021-12-02 08:15:53,832 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:15:53,835 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:15:53,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:15:54,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:15:54,278 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:15:54,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:15:54,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:15:54,318 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:15:54,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:15:54,359 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:15:54,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:15:54,390 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:15:54,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:15:54,428 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 08:15:54,454 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:15:54,474 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:54,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:54,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:15:54,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:15:54,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:54,617 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:54,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:15:54,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:15:54,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:54,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:54,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:15:54,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:15:54,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:54,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:54,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:15:54,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:15:54,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:54,852 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:54,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:15:54,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:15:54,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:54,933 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:54,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:15:54,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:15:54,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:15:54,986 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 08:15:54,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:15:54,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:55,012 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:15:55,023 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 08:15:55,023 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:15:55,026 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:15:55,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:15:55,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:15:55,532 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:15:55,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:15:55,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:15:55,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:15:55,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:15:55,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:15:55,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:15:55,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:15:55,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:15:55,683 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 08:15:55,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:15:55,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:55,785 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:55,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:15:55,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:15:55,895 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:55,931 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:55,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:15:55,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:15:56,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:56,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:56,061 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:15:56,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:15:56,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:56,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:56,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:15:56,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:15:56,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:15:56,158 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 08:15:56,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:15:56,170 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:56,183 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:15:56,194 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 08:15:56,194 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:15:56,197 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:15:56,311 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:15:56,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:15:56,648 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:15:56,650 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:15:56,670 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:15:56,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:15:56,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:15:56,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:15:56,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:15:56,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:15:56,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:15:56,793 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 08:15:56,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:15:56,846 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:56,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:56,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:15:56,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:15:57,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:57,068 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:57,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:15:57,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:15:57,201 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:57,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:57,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:15:57,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:15:57,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:57,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:57,456 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:15:57,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:15:57,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:57,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:57,579 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:15:57,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:15:57,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:57,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:57,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:15:57,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:15:57,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:57,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:57,790 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:15:57,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:15:57,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:57,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:57,885 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:15:57,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:15:57,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:57,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:57,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:15:58,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:15:58,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:15:58,059 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 08:15:58,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:15:58,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:58,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:15:58,109 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 08:15:58,109 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:15:58,112 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:15:58,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:15:58,520 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:15:58,601 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:15:58,602 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:15:58,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:15:58,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:15:58,658 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:15:58,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:15:58,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:15:58,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:15:58,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:15:58,737 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 08:15:58,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:15:58,785 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:58,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:58,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:15:58,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:15:58,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:58,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:59,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:15:59,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:15:59,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:59,140 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:59,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:15:59,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:15:59,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:59,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:59,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:15:59,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:15:59,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:59,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:15:59,456 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:15:59,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:15:59,507 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:15:59,523 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 08:15:59,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:15:59,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:15:59,558 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:15:59,572 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 08:15:59,572 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:15:59,572 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:15:59,575 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:15:59,688 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:16:00,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:16:00,499 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:16:00,499 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:16:00,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:16:00,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:16:00,551 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:16:00,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:16:00,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:16:00,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:16:00,617 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:00,776 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:16:00,782 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:00,783 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:02,783 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:02,940 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:16:02,944 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:02,946 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:04,946 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:05,115 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:16:05,119 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:05,119 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:07,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:07,292 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:16:07,296 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:07,296 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:09,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:09,463 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:16:09,470 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:09,470 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:11,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:11,644 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:16:11,650 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:11,650 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:13,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:13,822 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:16:13,828 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:13,828 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:15,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:15,994 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:16:16,000 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:16,000 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:18,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:18,147 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:18,147 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:20,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:20,295 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:20,295 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:22,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:22,457 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:22,457 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:24,459 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:24,605 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:24,605 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:26,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:26,775 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:26,775 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:28,775 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:28,920 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:28,920 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:30,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:31,065 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:31,065 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:33,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:33,217 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:33,217 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:35,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:35,361 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:35,361 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:37,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:37,562 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:37,562 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:39,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:39,769 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:39,769 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:41,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:41,947 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:41,947 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:43,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:44,153 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:44,154 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:46,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:46,345 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:46,345 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:48,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:48,562 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:48,562 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:50,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:50,733 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:50,733 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:52,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:52,907 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:52,907 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:54,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:55,105 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:55,107 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:57,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:57,319 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:57,321 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:16:59,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:16:59,525 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:16:59,525 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:01,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:01,767 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:01,769 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:03,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:03,934 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:03,934 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:05,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:06,101 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:06,101 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:08,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:08,263 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:08,263 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:10,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:10,424 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:10,424 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:12,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:12,586 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:12,588 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:14,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:14,753 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:14,753 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:16,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:16,934 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:16,934 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:18,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:19,111 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:19,111 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:21,112 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:21,276 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:21,276 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:23,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:23,440 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:23,440 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:25,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:25,596 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:25,596 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:27,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:27,767 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:27,767 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:29,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:29,938 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:29,938 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:31,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:32,111 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:32,111 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:34,112 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:34,275 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:34,275 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:36,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:36,450 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:36,450 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:38,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:38,609 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:38,609 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:40,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:40,788 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:40,788 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:42,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:42,953 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:42,953 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:44,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:45,118 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:45,118 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:47,119 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:47,295 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:17:47,295 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:17:49,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:17:49,513 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:17:49,515 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 109.942000151. 2021-12-02 08:17:49,515 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:17:49,517 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:17:49,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:18:00,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:18:00,578 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:18:00,579 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:18:00,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:18:00,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:18:00,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:18:00,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:18:00,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:18:00,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:18:00,734 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:18:00,892 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:18:00,892 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 11.3779997826. 2021-12-02 08:18:00,894 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:18:00,898 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:18:01,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:18:01,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:18:01,411 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:18:01,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:18:01,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:18:01,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:18:01,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:18:01,512 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:18:01,530 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:18:01,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:18:01,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:18:01,755 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:18:01,755 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.861000061035. 2021-12-02 08:18:01,756 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:18:01,757 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:18:01,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:18:02,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:18:02,200 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:18:02,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:18:02,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:18:02,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:18:02,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:18:02,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:18:02,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:18:02,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:18:02,341 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:18:02,516 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:18:02,516 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.759999990463. ############################################################ # # RAM FootPrint: 5823.3984375 # ############################################################ 2021-12-02 08:18:02,548 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 08:18:02,548 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 83 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 2021-12-02 08:19:25,552 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 08:19:25,552 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 08:19:30,555 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:19:30,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:19:31,178 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:19:31,262 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:19:31,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:19:31,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:19:31,295 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:19:31,313 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:19:31,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:19:31,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:19:31,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:19:31,421 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:19:31,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:19:31,551 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['26588 memcached\n'], []) 2021-12-02 08:19:31,551 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 08:19:31,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:19:31,719 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['26588 memcached\n'], []) 2021-12-02 08:19:31,720 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:19:31,723 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:19:31,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:19:32,127 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:19:32,269 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:19:32,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:19:32,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:19:32,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:19:32,319 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:19:32,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:19:32,395 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:19:32,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:19:32,431 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:19:32,431 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:19:32,526 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['105262 memcached\n'], []) 2021-12-02 08:19:32,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 08:19:32,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:19:32,734 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['105262 memcached\n'], []) 2021-12-02 08:19:32,734 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:19:32,737 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:19:32,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:19:33,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:19:33,233 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:19:33,234 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:19:33,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:19:33,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:19:33,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:19:33,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:19:33,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:19:33,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:19:33,378 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:19:33,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:19:33,467 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['229069 memcached\n'], []) 2021-12-02 08:19:33,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 08:19:33,548 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:19:33,618 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['229069 memcached\n'], []) 2021-12-02 08:19:33,619 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:19:33,621 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:19:33,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:19:33,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:19:34,094 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:19:34,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:19:34,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:19:34,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:19:34,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:19:34,206 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:19:34,232 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:19:34,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:19:34,282 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:19:34,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:19:34,377 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['13159 memcached\n'], []) 2021-12-02 08:19:34,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 08:19:34,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:19:34,530 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['13159 memcached\n'], []) 2021-12-02 08:19:34,532 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:19:34,535 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:19:34,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:19:34,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:19:34,969 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:19:34,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:19:34,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:19:35,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:19:35,020 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:19:35,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:19:35,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:19:35,063 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:19:35,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:19:35,091 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 08:19:35,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:19:35,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:35,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:35,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:19:35,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:19:35,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:35,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:35,352 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:19:35,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:19:35,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:35,438 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:35,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:19:35,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:19:35,520 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:35,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:35,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:19:35,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:19:35,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:35,635 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:35,670 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:19:35,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:19:35,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:35,749 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:35,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:19:35,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:19:35,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:19:35,818 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 08:19:35,818 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:19:35,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:35,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:19:35,861 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 08:19:35,861 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:19:35,864 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:19:35,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:19:36,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:19:36,352 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:19:36,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:19:36,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:19:36,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:19:36,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:19:36,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:19:36,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:19:36,454 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:19:36,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:19:36,484 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 08:19:36,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:19:36,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:36,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:36,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:19:36,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:19:36,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:36,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:36,734 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:19:36,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:19:36,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:36,798 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:36,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:19:36,849 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:19:36,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:36,885 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:36,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:19:36,915 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:19:36,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:19:36,944 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 08:19:36,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:19:36,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:36,971 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:19:36,984 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 08:19:36,984 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:19:36,986 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:19:37,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:19:37,313 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:19:37,398 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:19:37,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:19:37,418 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:19:37,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:19:37,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:19:37,477 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:19:37,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:19:37,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:19:37,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:19:37,559 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 08:19:37,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:19:37,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:37,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:37,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:19:37,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:19:37,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:37,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:37,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:19:37,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:19:37,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:37,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:38,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:19:38,053 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:19:38,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:38,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:38,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:19:38,186 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:19:38,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:38,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:38,262 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:19:38,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:19:38,303 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:38,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:38,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:19:38,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:19:38,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:38,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:38,436 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:19:38,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:19:38,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:38,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:38,545 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:19:38,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:19:38,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:38,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:38,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:19:38,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:19:38,709 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:19:38,726 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 08:19:38,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:19:38,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:38,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:19:38,772 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 08:19:38,773 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:19:38,775 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:19:38,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:19:39,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:19:39,253 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:19:39,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:19:39,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:19:39,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:19:39,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:19:39,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:19:39,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:19:39,367 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:19:39,382 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:19:39,398 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 08:19:39,426 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:19:39,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:39,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:39,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:19:39,571 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:19:39,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:39,642 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:39,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:19:39,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:19:39,762 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:39,808 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:39,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:19:39,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:19:39,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:39,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:39,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:19:40,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:19:40,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:40,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:19:40,102 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:19:40,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:19:40,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:19:40,158 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 08:19:40,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:19:40,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:19:40,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:19:40,198 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 08:19:40,200 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:19:40,200 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:19:40,201 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:19:40,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:19:41,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:19:41,125 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:19:41,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:19:41,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:19:41,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:19:41,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:19:41,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:19:41,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:19:41,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:19:41,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:19:41,404 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:19:41,411 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:19:41,411 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:19:43,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:19:43,555 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:19:43,559 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:19:43,559 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:19:45,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:19:45,733 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:19:45,737 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:19:45,737 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:19:47,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:19:47,900 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:19:47,905 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:19:47,905 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:19:49,905 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:19:50,065 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:19:50,069 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:19:50,069 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:19:52,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:19:52,181 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:19:52,187 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:19:52,187 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:19:54,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:19:54,352 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:19:54,358 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:19:54,358 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:19:56,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:19:56,529 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:19:56,532 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:19:56,533 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:19:58,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:19:58,720 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:19:58,726 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:19:58,726 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:00,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:00,874 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:00,874 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:02,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:03,019 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:03,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:05,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:05,176 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:05,176 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:07,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:07,322 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:07,322 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:09,323 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:09,469 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:09,470 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:11,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:11,621 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:11,621 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:13,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:13,766 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:13,766 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:15,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:15,931 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:15,931 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:17,931 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:18,157 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:18,157 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:20,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:20,349 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:20,349 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:22,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:22,571 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:22,571 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:24,571 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:24,789 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:24,789 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:26,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:27,013 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:27,013 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:29,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:29,211 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:29,211 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:31,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:31,447 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:31,447 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:33,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:33,641 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:33,641 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:35,642 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:35,836 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:35,838 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:37,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:38,040 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:38,042 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:40,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:40,249 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:40,250 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:42,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:42,459 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:42,459 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:44,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:44,625 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:44,627 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:46,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:46,782 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:46,782 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:48,783 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:48,944 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:48,944 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:50,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:51,107 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:51,107 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:53,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:53,266 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:53,266 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:55,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:55,424 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:55,424 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:57,426 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:57,575 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:57,575 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:20:59,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:20:59,733 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:20:59,733 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:01,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:01,911 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:01,911 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:03,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:04,082 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:04,082 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:06,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:06,246 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:06,246 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:08,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:08,411 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:08,411 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:10,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:10,582 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:10,582 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:12,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:12,744 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:12,744 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:14,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:14,907 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:14,907 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:16,908 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:17,096 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:17,096 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:19,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:19,282 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:19,283 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:21,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:21,467 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:21,467 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:23,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:23,635 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:23,635 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:25,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:25,813 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:25,813 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:27,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:27,967 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:27,969 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:29,969 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:30,134 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:30,134 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:32,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:32,305 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:21:32,306 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:21:34,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:34,476 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:21:34,476 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 114.276000023. 2021-12-02 08:21:34,477 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:21:34,479 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:21:34,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:21:34,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:21:34,921 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:21:34,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:21:34,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:21:34,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:21:34,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:21:35,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:21:35,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:21:35,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:21:35,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:35,257 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:21:35,259 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.78200006485. 2021-12-02 08:21:35,259 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:21:35,262 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:21:35,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:21:35,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:21:35,739 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:21:35,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:21:35,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:21:35,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:21:35,806 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:21:35,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:21:35,846 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:21:35,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:21:35,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:36,071 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:21:36,071 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.81200003624. 2021-12-02 08:21:36,072 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:21:36,073 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:21:36,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:21:36,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:21:36,553 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:21:36,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:21:36,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:21:36,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:21:36,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:21:36,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:21:36,668 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:21:36,686 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:21:36,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:21:36,880 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:21:36,880 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.808000087738. ############################################################ # # RAM FootPrint: 5802.5546875 # ############################################################ 2021-12-02 08:21:36,910 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 08:21:36,910 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 71 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 2021-12-02 08:22:47,937 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 08:22:47,937 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 08:22:52,938 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:22:53,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:22:54,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:22:54,440 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:22:54,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:22:54,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:22:54,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:22:54,556 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:22:54,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:22:54,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:22:54,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:22:54,700 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:22:54,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:22:54,786 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['28121 memcached\n'], []) 2021-12-02 08:22:54,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 08:22:54,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:22:54,917 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['28121 memcached\n'], []) 2021-12-02 08:22:54,918 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:22:54,921 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:22:55,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:22:55,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:22:56,015 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:22:56,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:22:56,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:22:56,053 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:22:56,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:22:56,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:22:56,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:22:56,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:22:56,153 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:22:56,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:22:56,233 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['106150 memcached\n'], []) 2021-12-02 08:22:56,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 08:22:56,318 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:22:56,378 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['106150 memcached\n'], []) 2021-12-02 08:22:56,378 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:22:56,381 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:22:56,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:22:56,783 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:22:56,877 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:22:56,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:22:56,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:22:56,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:22:56,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:22:56,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:22:56,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:22:57,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:22:57,038 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:22:57,038 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:22:57,112 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['229941 memcached\n'], []) 2021-12-02 08:22:57,112 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 08:22:57,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:22:57,214 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['229941 memcached\n'], []) 2021-12-02 08:22:57,216 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:22:57,219 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:22:57,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:22:57,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:22:57,686 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:22:57,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:22:57,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:22:57,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:22:57,734 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:22:57,752 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:22:57,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:22:57,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:22:57,796 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:22:57,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:22:57,842 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['14325 memcached\n'], []) 2021-12-02 08:22:57,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 08:22:57,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:22:58,009 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['14325 memcached\n'], []) 2021-12-02 08:22:58,009 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:22:58,015 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:22:58,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:22:58,392 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:22:58,463 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:22:58,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:22:58,482 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:22:58,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:22:58,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:22:58,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:22:58,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:22:58,563 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:22:58,579 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:22:58,595 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 08:22:58,622 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:22:58,642 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:22:58,678 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:22:58,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:22:58,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:22:58,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:22:58,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:22:58,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:22:58,903 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:22:58,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:22:58,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:22:59,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:22:59,073 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:22:59,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:22:59,147 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:22:59,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:22:59,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:22:59,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:22:59,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:22:59,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:22:59,349 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:22:59,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:22:59,400 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:22:59,423 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:22:59,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:22:59,459 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:22:59,473 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 08:22:59,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:22:59,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:22:59,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:22:59,519 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 08:22:59,519 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:22:59,523 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:22:59,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:23:04,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:23:05,036 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:23:05,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:23:05,053 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:23:05,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:23:05,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:23:05,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:23:05,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:23:05,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:23:05,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:23:05,173 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 08:23:05,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:23:05,224 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:05,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:05,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:23:05,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:23:05,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:05,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:05,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:23:05,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:23:05,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:05,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:05,591 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:23:05,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:23:05,653 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:05,678 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:05,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:23:05,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:23:05,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:23:05,779 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 08:23:05,780 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:23:05,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:05,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:23:05,829 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 08:23:05,831 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:23:05,834 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:23:05,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:23:06,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:23:06,302 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:23:06,303 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:23:06,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:23:06,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:23:06,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:23:06,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:23:06,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:23:06,418 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:23:06,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:23:06,453 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 08:23:06,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:23:06,505 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:06,551 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:06,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:23:06,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:23:06,670 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:06,719 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:06,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:23:06,818 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:23:06,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:06,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:06,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:23:07,010 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:23:07,045 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:07,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:07,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:23:07,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:23:07,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:07,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:07,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:23:07,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:23:07,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:07,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:07,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:23:07,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:23:07,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:07,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:07,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:23:07,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:23:07,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:07,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:07,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:23:07,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:23:07,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:07,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:07,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:23:07,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:23:07,946 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:23:07,970 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 08:23:07,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:23:07,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:08,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:23:08,039 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 08:23:08,040 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:23:08,046 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:23:08,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:23:08,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:23:08,434 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:23:08,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:23:08,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:23:08,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:23:08,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:23:08,520 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:23:08,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:23:08,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:23:08,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:23:08,595 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 08:23:08,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:23:08,650 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:08,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:08,749 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:23:08,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:23:08,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:08,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:08,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:23:08,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:23:08,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:09,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:09,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:23:09,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:23:09,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:09,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:09,224 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:23:09,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:23:09,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:09,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:09,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:23:09,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:23:09,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:09,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:23:09,454 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:23:09,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:23:09,492 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:23:09,507 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 08:23:09,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:23:09,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:23:09,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:23:09,555 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 08:23:09,555 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:23:09,555 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:23:09,558 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:23:09,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:23:09,933 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:23:10,017 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:23:10,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:23:10,038 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:23:10,056 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:23:10,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:23:10,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:23:10,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:23:10,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:23:10,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:10,280 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:23:10,286 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:10,286 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:12,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:12,450 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:23:12,457 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:12,457 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:14,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:14,638 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:23:14,642 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:14,642 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:16,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:16,805 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:23:16,809 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:16,811 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:18,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:18,973 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:23:18,979 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:18,979 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:20,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:21,145 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:23:21,150 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:21,151 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:23,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:23,382 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:23,384 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:25,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:25,529 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:25,529 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:27,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:27,678 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:27,678 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:29,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:29,831 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:29,831 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:31,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:32,052 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:32,052 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:34,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:34,206 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:34,206 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:36,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:36,371 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:36,372 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:38,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:38,516 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:38,516 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:40,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:40,673 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:40,674 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:42,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:42,869 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:42,869 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:44,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:45,073 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:45,075 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:47,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:47,302 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:47,302 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:49,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:49,516 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:49,516 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:51,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:51,691 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:51,691 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:53,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:53,915 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:53,917 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:55,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:56,088 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:56,089 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:23:58,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:23:58,288 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:23:58,288 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:00,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:00,516 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:00,516 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:02,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:02,703 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:02,703 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:04,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:04,875 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:04,875 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:06,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:07,079 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:07,079 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:09,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:09,404 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:09,404 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:11,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:11,573 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:11,573 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:13,573 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:13,724 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:13,724 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:15,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:15,878 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:15,878 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:17,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:18,040 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:18,040 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:20,040 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:20,198 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:20,198 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:22,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:22,368 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:22,368 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:24,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:24,576 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:24,578 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:26,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:26,733 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:26,733 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:28,734 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:28,920 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:28,920 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:30,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:31,088 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:31,089 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:33,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:33,269 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:33,269 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:35,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:35,446 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:35,447 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:37,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:37,624 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:37,625 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:39,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:39,788 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:39,788 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:41,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:41,950 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:41,950 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:43,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:44,108 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:44,108 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:46,109 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:46,286 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:46,286 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:48,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:48,457 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:48,457 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:50,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:50,627 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:50,627 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:52,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:52,801 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:52,801 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:54,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:54,976 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:54,977 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:56,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:57,158 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:24:57,158 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:24:59,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:24:59,523 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:24:59,523 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 109.967999935. 2021-12-02 08:24:59,523 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:24:59,526 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:24:59,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:24:59,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:25:00,049 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:25:00,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:25:00,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:25:00,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:25:00,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:25:00,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:25:00,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:25:00,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:25:00,326 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:25:00,463 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:25:00,463 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.9390001297. 2021-12-02 08:25:00,463 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:25:00,466 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:25:00,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:25:00,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:25:00,953 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:25:00,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:25:00,971 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:25:00,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:25:01,022 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:25:01,048 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:25:01,068 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:25:01,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:25:01,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:25:01,335 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:25:01,335 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.871999979019. 2021-12-02 08:25:01,335 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:25:01,338 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:25:01,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:25:01,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:25:01,805 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:25:01,806 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:25:01,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:25:01,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:25:01,882 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:25:01,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:25:01,928 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:25:01,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:25:01,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:25:02,138 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:25:02,140 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.805000066757. ############################################################ # # RAM FootPrint: 5777.734375 # ############################################################ 2021-12-02 08:25:02,171 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 08:25:02,171 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 75 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 2021-12-02 08:26:17,176 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 08:26:17,177 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 08:26:22,178 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:26:22,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:26:22,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:26:22,915 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:26:22,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:26:22,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:26:23,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:26:23,137 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:26:23,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:26:23,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:26:23,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:26:23,272 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:26:23,272 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:26:23,344 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['29646 memcached\n'], []) 2021-12-02 08:26:23,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 08:26:23,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:26:23,569 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['29646 memcached\n'], []) 2021-12-02 08:26:23,569 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:26:23,572 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:26:23,701 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:26:23,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:26:24,082 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:26:24,084 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:26:24,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:26:24,130 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:26:24,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:26:24,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:26:24,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:26:24,214 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:26:24,232 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:26:24,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:26:24,293 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['106782 memcached\n'], []) 2021-12-02 08:26:24,295 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 08:26:24,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:26:24,447 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['106782 memcached\n'], []) 2021-12-02 08:26:24,448 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:26:24,450 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:26:24,581 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:26:24,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:26:24,964 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:26:24,966 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:26:24,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:26:25,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:26:25,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:26:25,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:26:25,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:26:25,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:26:25,115 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:26:25,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:26:25,191 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['230798 memcached\n'], []) 2021-12-02 08:26:25,191 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 08:26:25,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:26:25,313 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['230798 memcached\n'], []) 2021-12-02 08:26:25,315 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:26:25,318 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:26:25,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:26:25,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:26:25,786 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:26:25,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:26:25,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:26:25,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:26:25,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:26:25,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:26:25,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:26:25,905 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:26:25,926 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:26:25,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:26:25,983 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['15347 memcached\n'], []) 2021-12-02 08:26:25,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 08:26:26,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:26:26,144 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['15347 memcached\n'], []) 2021-12-02 08:26:26,144 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:26:26,148 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:26:26,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:26:26,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:26:26,598 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:26:26,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:26:26,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:26:26,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:26:26,655 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:26:26,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:26:26,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:26:26,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:26:26,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:26:26,732 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 08:26:26,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:26:26,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:26,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:26,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:26:26,905 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:26:26,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:26,987 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:27,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:26:27,073 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:26:27,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:27,150 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:27,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:26:27,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:26:27,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:27,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:27,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:26:27,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:26:27,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:27,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:27,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:26:27,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:26:27,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:27,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:27,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:26:27,630 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:26:27,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:26:27,671 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 08:26:27,673 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:26:27,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:27,709 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:26:27,724 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 08:26:27,726 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:26:27,734 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:26:27,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:26:28,137 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:26:28,216 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:26:28,216 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:26:28,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:26:28,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:26:28,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:26:28,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:26:28,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:26:28,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:26:28,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:26:28,348 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 08:26:28,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:26:28,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:28,441 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:28,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:26:28,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:26:28,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:28,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:28,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:26:28,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:26:28,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:28,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:28,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:26:28,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:26:28,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:28,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:28,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:26:28,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:26:28,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:26:28,839 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 08:26:28,839 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:26:28,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:28,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:26:28,885 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 08:26:28,887 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:26:28,888 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:26:29,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:26:29,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:26:29,381 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:26:29,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:26:29,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:26:29,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:26:29,438 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:26:29,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:26:29,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:26:29,489 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:26:29,505 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:26:29,523 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 08:26:29,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:26:29,581 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:29,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:29,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:26:29,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:26:29,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:29,785 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:29,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:26:29,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:26:29,905 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:29,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:30,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:26:30,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:26:30,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:30,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:30,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:26:30,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:26:30,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:30,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:30,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:26:30,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:26:30,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:30,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:30,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:26:30,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:26:30,469 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:30,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:30,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:26:30,545 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:26:30,563 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:30,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:30,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:26:30,630 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:26:30,647 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:30,668 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:30,691 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:26:30,709 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:26:30,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:26:30,744 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 08:26:30,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:26:30,762 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:30,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:26:30,792 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 08:26:30,792 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:26:30,795 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:26:30,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:26:31,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:26:31,266 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:26:31,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:26:31,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:26:31,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:26:31,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:26:31,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:26:31,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:26:31,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:26:31,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:26:31,384 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 08:26:31,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:26:31,433 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:31,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:31,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:26:31,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:26:31,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:31,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:31,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:26:31,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:26:31,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:31,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:31,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:26:31,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:26:31,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:31,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:31,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:26:32,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:26:32,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:32,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:32,112 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:26:32,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:26:32,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:32,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:26:32,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:26:32,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:26:32,242 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:26:32,256 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 08:26:32,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:26:32,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:26:32,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:26:32,299 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 08:26:32,299 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:26:32,301 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:26:32,302 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:26:32,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:26:32,673 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:26:32,743 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:26:32,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:26:32,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:26:32,778 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:26:32,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:26:32,821 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:26:32,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:26:32,849 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:26:32,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:32,966 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:26:32,970 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:32,970 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:26:34,971 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:35,075 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:26:35,079 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:35,081 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:26:37,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:37,242 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:26:37,247 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:37,247 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:26:39,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:39,414 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:26:39,418 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:39,418 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:26:41,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:41,595 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:26:41,599 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:41,599 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:26:43,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:43,779 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:26:43,786 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:43,786 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:26:45,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:45,970 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:26:45,976 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:45,976 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:26:47,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:48,118 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:48,119 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:26:50,119 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:50,263 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:50,263 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:26:52,265 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:52,414 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:52,415 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:26:54,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:54,562 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:54,562 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:26:56,563 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:56,710 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:56,710 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:26:58,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:26:58,858 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:26:58,858 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:00,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:00,999 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:00,999 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:02,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:03,138 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:03,138 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:05,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:05,365 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:05,367 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:07,367 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:07,594 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:07,594 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:09,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:09,818 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:09,818 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:11,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:12,046 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:12,046 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:14,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:14,246 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:14,246 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:16,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:16,448 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:16,450 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:18,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:18,605 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:18,605 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:20,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:20,851 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:20,851 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:22,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:23,019 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:23,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:25,020 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:25,176 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:25,176 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:27,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:27,339 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:27,339 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:29,341 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:29,503 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:29,503 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:31,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:31,667 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:31,667 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:33,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:33,832 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:33,832 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:35,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:36,003 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:36,003 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:38,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:38,165 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:38,167 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:40,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:40,318 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:40,319 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:42,319 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:42,480 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:42,480 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:44,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:44,644 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:44,644 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:46,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:46,819 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:46,819 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:48,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:48,983 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:48,983 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:50,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:51,145 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:51,145 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:53,147 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:53,308 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:53,309 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:55,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:55,474 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:55,476 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:57,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:57,632 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:57,632 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:27:59,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:27:59,792 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:27:59,792 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:28:01,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:28:01,959 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:28:01,960 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:28:03,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:28:04,130 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:28:04,131 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:28:06,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:28:06,295 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:28:06,296 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:28:08,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:28:08,476 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:28:08,476 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:28:10,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:28:10,651 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:28:10,651 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:28:12,653 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:28:12,901 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:28:12,901 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 100.599999905. 2021-12-02 08:28:12,901 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:28:12,904 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:28:13,058 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:28:13,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:28:13,894 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:28:13,895 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:28:13,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:28:13,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:28:14,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:28:14,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:28:14,229 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:28:14,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:28:14,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:28:14,549 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:28:14,549 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-02 08:28:16,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:28:16,809 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:28:16,809 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 3.90799999237. 2021-12-02 08:28:16,811 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:28:16,812 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:28:16,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:28:17,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:28:17,267 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:28:17,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:28:17,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:28:17,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:28:17,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:28:17,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:28:17,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:28:17,397 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:28:17,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:28:17,611 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:28:17,611 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.801000118256. 2021-12-02 08:28:17,611 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:28:17,617 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:28:17,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:28:17,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:28:17,967 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:28:17,969 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:28:17,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:28:18,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:28:18,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:28:18,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:28:18,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:28:18,068 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:28:18,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:28:18,246 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:28:18,246 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.634999990463. ############################################################ # # RAM FootPrint: 5730.38671875 # ############################################################ 2021-12-02 08:28:18,276 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 08:28:18,276 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 63 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 2021-12-02 08:29:21,279 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 08:29:21,280 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 08:29:26,282 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:29:26,456 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:29:26,959 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:29:27,036 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:29:27,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:29:27,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:29:27,073 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:29:27,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:29:27,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:29:27,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:29:27,201 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:29:27,219 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:29:27,219 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:29:27,349 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['31166 memcached\n'], []) 2021-12-02 08:29:27,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 08:29:27,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:29:27,509 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['31166 memcached\n'], []) 2021-12-02 08:29:27,509 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:29:27,512 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:29:27,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:29:27,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:29:28,059 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:29:28,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:29:28,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:29:28,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:29:28,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:29:28,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:29:28,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:29:28,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:29:28,259 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:29:28,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:29:28,329 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['107953 memcached\n'], []) 2021-12-02 08:29:28,329 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 08:29:28,410 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:29:28,500 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['107953 memcached\n'], []) 2021-12-02 08:29:28,500 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:29:28,503 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:29:28,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:29:28,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:29:29,010 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:29:29,012 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:29:29,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:29:29,045 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:29:29,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:29:29,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:29:29,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:29:29,127 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:29:29,144 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:29:29,147 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:29:29,190 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['231656 memcached\n'], []) 2021-12-02 08:29:29,191 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 08:29:29,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:29:29,299 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['231656 memcached\n'], []) 2021-12-02 08:29:29,299 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:29:29,302 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:29:29,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:29:30,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:29:30,135 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:29:30,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:29:30,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:29:30,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:29:30,191 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:29:30,209 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:29:30,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:29:30,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:29:30,266 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:29:30,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:29:30,335 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['16657 memcached\n'], []) 2021-12-02 08:29:30,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 08:29:30,456 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:29:30,634 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['16657 memcached\n'], []) 2021-12-02 08:29:30,634 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:29:30,640 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:29:30,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:29:31,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:29:31,098 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:29:31,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:29:31,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:29:31,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:29:31,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:29:31,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:29:31,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:29:31,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:29:31,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:29:31,236 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 08:29:31,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:29:31,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:31,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:31,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:29:31,436 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:29:31,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:31,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:31,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:29:31,596 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:29:31,630 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:31,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:31,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:29:31,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:29:31,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:31,841 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:31,885 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:29:31,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:29:31,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:31,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:32,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:29:32,040 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:29:32,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:32,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:32,109 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:29:32,127 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:29:32,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:29:32,157 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 08:29:32,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:29:32,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:32,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:29:32,200 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 08:29:32,200 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:29:32,203 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:29:32,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:29:32,596 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:29:32,670 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:29:32,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:29:32,688 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:29:32,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:29:32,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:29:32,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:29:32,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:29:32,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:29:32,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:29:32,803 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 08:29:32,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:29:32,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:32,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:32,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:29:33,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:29:33,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:33,076 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:33,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:29:33,147 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:29:33,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:33,209 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:33,247 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:29:33,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:29:33,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:33,336 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:33,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:29:33,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:29:33,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:29:33,436 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 08:29:33,436 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:29:33,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:33,469 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:29:33,483 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 08:29:33,484 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:29:33,487 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:29:33,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:29:33,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:29:33,951 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:29:33,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:29:33,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:29:33,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:29:34,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:29:34,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:29:34,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:29:34,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:29:34,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:29:34,109 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 08:29:34,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:29:34,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:34,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:34,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:29:34,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:29:34,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:34,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:34,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:29:34,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:29:34,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:34,548 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:34,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:29:34,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:29:34,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:34,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:34,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:29:34,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:29:34,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:34,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:34,880 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:29:34,908 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:29:34,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:34,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:35,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:29:35,038 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:29:35,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:35,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:35,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:29:35,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:29:35,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:35,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:35,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:29:35,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:29:35,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:35,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:35,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:29:35,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:29:35,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:35,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:35,545 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-02 08:29:35,563 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-02 08:29:35,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:29:35,598 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 08:29:35,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:29:35,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:35,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:29:35,650 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 08:29:35,651 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:29:35,654 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:29:35,762 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:29:36,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:29:36,088 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:29:36,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:29:36,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:29:36,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:29:36,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:29:36,168 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:29:36,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:29:36,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:29:36,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:29:36,240 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 08:29:36,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:29:36,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:36,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:36,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:29:36,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:29:36,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:36,436 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:36,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:29:36,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:29:36,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:36,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:36,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:29:36,673 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:29:36,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:36,742 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:36,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:29:36,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:29:36,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:36,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:36,910 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:29:36,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:29:36,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:36,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:29:36,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:29:37,012 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:29:37,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:29:37,042 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 08:29:37,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:29:37,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:29:37,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:29:37,082 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 08:29:37,082 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:29:37,082 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:29:37,084 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:29:37,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:29:37,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:29:37,506 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:29:37,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:29:37,528 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:29:37,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:29:37,576 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:29:37,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:29:37,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:29:37,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:29:37,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:29:37,813 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:29:37,819 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:29:37,819 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:29:39,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:29:39,967 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:29:39,973 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:29:39,973 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:29:41,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:29:42,144 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:29:42,148 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:29:42,148 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:29:44,150 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:29:44,322 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:29:44,326 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:29:44,326 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:29:46,326 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:29:46,496 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:29:46,502 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:29:46,502 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:29:48,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:29:48,667 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:29:48,673 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:29:48,673 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:29:50,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:29:50,845 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:29:50,851 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:29:50,851 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:29:52,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:29:53,006 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:29:53,006 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:29:55,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:29:55,148 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:29:55,148 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:29:57,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:29:57,296 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:29:57,296 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:29:59,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:29:59,437 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:29:59,438 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:01,438 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:01,595 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:01,596 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:03,596 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:03,744 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:03,746 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:05,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:05,887 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:05,887 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:07,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:08,108 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:08,108 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:10,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:10,338 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:10,338 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:12,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:12,558 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:12,559 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:14,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:14,760 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:14,762 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:16,762 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:16,973 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:16,973 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:18,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:19,155 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:19,155 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:21,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:21,361 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:21,361 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:23,362 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:23,562 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:23,562 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:25,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:25,717 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:25,717 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:27,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:27,859 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:27,859 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:29,859 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:30,017 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:30,017 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:32,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:32,174 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:32,176 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:34,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:34,349 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:34,349 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:36,349 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:36,503 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:36,503 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:38,505 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:38,661 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:38,661 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:40,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:40,815 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:40,815 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:42,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:42,974 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:42,976 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:44,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:45,140 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:45,140 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:47,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:47,301 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:47,301 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:49,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:49,461 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:49,461 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:51,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:51,642 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:51,642 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:53,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:53,819 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:53,819 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:55,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:55,997 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:55,997 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:30:57,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:30:58,190 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:30:58,190 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:31:00,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:00,364 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:00,364 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:31:02,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:02,549 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:02,551 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:31:04,551 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:04,733 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:04,733 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:31:06,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:06,894 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:06,894 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:31:08,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:09,065 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:09,066 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:31:11,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:11,246 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:11,246 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:31:13,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:13,410 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:13,410 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:31:15,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:15,565 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:15,565 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:31:17,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:18,006 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:31:18,007 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 100.92599988. 2021-12-02 08:31:18,009 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:31:18,010 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:31:18,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:31:18,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:31:18,607 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:31:18,607 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:31:18,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:31:18,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:31:18,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:31:18,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:31:18,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:31:18,969 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:31:18,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:19,234 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:31:19,234 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.22599983215. 2021-12-02 08:31:19,234 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:31:19,237 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:31:19,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:31:19,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:31:19,714 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:31:19,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:31:19,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:31:19,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:31:19,778 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:31:19,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:31:19,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:31:19,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:31:19,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:20,078 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:31:20,079 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.84500002861. 2021-12-02 08:31:20,081 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:31:20,082 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:31:20,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:31:20,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:31:20,423 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:31:20,423 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:31:20,438 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:31:20,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:31:20,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:31:20,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:31:20,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:31:20,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:31:20,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:20,703 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:20,703 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-02 08:31:22,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:22,864 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:22,864 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-02 08:31:24,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:25,026 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:25,026 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-02 08:31:27,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:27,194 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:27,194 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-02 08:31:29,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:29,361 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:31:29,361 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-02 08:31:31,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:31:31,638 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:31:31,638 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 11.5569999218. ############################################################ # # RAM FootPrint: 5695.546875 # ############################################################ 2021-12-02 08:31:31,667 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 08:31:31,667 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 105 seconds. Reason: Iteration:19 waiting to kill memc on all nodes 2021-12-02 08:33:16,673 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 08:33:16,674 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 08:33:21,676 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:33:21,844 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:33:22,137 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:33:22,219 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:33:22,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:33:22,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:33:22,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:33:22,278 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:33:22,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:33:22,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:33:22,329 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:33:22,349 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:33:22,349 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:33:22,408 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['32613 memcached\n'], []) 2021-12-02 08:33:22,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 08:33:22,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:33:22,532 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['32613 memcached\n'], []) 2021-12-02 08:33:22,532 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:33:22,535 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:33:22,688 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:33:23,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:33:23,088 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:33:23,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:33:23,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:33:23,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:33:23,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:33:23,201 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:33:23,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:33:23,239 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:33:23,257 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:33:23,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:33:23,336 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['108860 memcached\n'], []) 2021-12-02 08:33:23,336 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 08:33:23,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:33:23,575 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['108860 memcached\n'], []) 2021-12-02 08:33:23,576 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:33:23,579 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:33:23,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:33:23,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:33:24,079 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:33:24,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:33:24,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:33:24,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:33:24,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:33:24,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:33:24,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:33:24,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:33:24,236 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:33:24,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:33:24,316 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['232478 memcached\n'], []) 2021-12-02 08:33:24,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 08:33:24,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:33:24,428 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['232478 memcached\n'], []) 2021-12-02 08:33:24,428 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:33:24,431 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:33:24,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:33:24,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:33:24,894 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:33:24,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:33:24,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:33:24,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:33:24,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:33:25,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:33:25,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:33:25,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:33:25,157 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:33:25,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:33:25,217 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['17821 memcached\n'], []) 2021-12-02 08:33:25,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 08:33:25,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:33:25,392 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['17821 memcached\n'], []) 2021-12-02 08:33:25,394 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:33:25,395 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:33:25,517 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:33:25,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:33:25,861 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:33:25,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:33:25,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:33:25,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:33:25,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:33:25,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:33:25,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:33:25,966 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:33:25,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:33:26,002 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 08:33:26,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:33:26,048 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:26,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:26,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:33:26,178 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:33:26,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:26,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:26,313 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:33:26,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:33:26,387 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:26,438 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:26,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:33:26,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:33:26,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:26,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:26,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:33:26,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:33:26,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:26,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:26,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:33:26,793 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:33:26,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:26,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:26,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:33:26,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:33:26,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:33:26,927 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 08:33:26,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:33:26,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:26,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:33:26,969 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 08:33:26,969 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:33:26,971 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:33:27,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:33:27,382 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:33:27,451 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:33:27,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:33:27,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:33:27,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:33:27,505 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:33:27,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:33:27,538 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:33:27,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:33:27,568 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:33:27,584 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 08:33:27,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:33:27,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:27,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:27,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:33:27,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:33:27,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:27,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:27,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:33:27,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:33:27,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:27,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:27,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:33:28,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:33:28,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:28,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:28,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:33:28,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:33:28,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:33:28,174 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 08:33:28,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:33:28,191 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:28,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:33:28,226 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 08:33:28,226 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:33:28,230 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:33:28,341 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:33:28,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:33:28,673 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:33:28,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:33:28,691 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:33:28,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:33:28,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:33:28,747 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:33:28,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:33:28,778 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:33:28,793 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:33:28,809 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 08:33:28,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:33:28,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:28,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:28,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:33:28,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:33:29,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:29,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:29,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:33:29,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:33:29,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:29,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:29,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:33:29,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:33:29,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:29,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:29,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:33:29,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:33:29,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:29,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:29,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:33:29,535 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:33:29,553 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:29,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:29,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:33:29,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:33:29,678 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:29,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:29,732 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:33:29,752 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:33:29,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:29,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:29,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:33:29,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:33:29,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:29,877 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:29,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:33:29,915 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:33:29,933 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:29,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:29,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-02 08:33:29,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-02 08:33:29,992 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:33:30,005 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 08:33:30,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:33:30,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:30,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:33:30,046 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 08:33:30,046 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:33:30,049 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:33:30,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:33:30,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:33:30,536 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:33:30,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:33:30,556 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:33:30,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:33:30,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:33:30,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:33:30,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:33:30,650 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:33:30,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:33:30,680 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 08:33:30,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:33:30,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:30,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:30,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:33:30,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:33:30,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:30,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:30,984 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:33:31,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:33:31,051 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:31,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:31,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:33:31,178 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:33:31,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:31,247 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:31,288 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:33:31,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:33:31,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:31,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:31,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:33:31,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:33:31,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:31,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:33:31,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:33:31,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:33:31,540 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:33:31,553 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 08:33:31,553 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:33:31,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:33:31,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:33:31,594 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 08:33:31,594 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:33:31,594 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:33:31,596 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:33:31,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:33:32,010 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:33:32,094 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:33:32,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:33:32,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:33:32,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:33:32,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:33:32,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:33:32,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:33:32,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:33:32,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:32,467 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:33:32,471 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:32,473 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:33:34,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:34,631 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:33:34,635 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:34,635 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:33:36,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:36,801 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:33:36,805 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:36,805 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:33:38,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:38,980 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:33:39,448 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:39,448 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:33:41,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:41,615 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:33:41,621 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:41,621 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:33:43,622 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:43,805 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:33:43,811 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:43,811 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:33:45,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:45,986 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:33:45,990 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:45,990 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:33:47,992 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:48,137 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:48,137 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:33:50,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:50,299 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:50,299 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:33:52,301 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:52,446 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:52,446 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:33:54,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:54,602 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:54,604 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:33:56,604 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:56,744 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:56,746 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:33:58,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:33:58,897 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:33:58,897 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:00,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:01,042 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:01,042 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:03,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:03,230 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:03,230 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:05,232 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:05,374 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:05,374 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:07,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:07,581 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:07,581 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:09,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:09,786 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:09,786 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:11,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:12,003 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:12,005 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:14,005 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:14,196 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:14,197 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:16,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:16,400 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:16,400 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:18,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:18,536 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:18,536 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:20,538 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:20,737 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:20,739 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:22,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:22,941 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:22,941 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:24,943 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:25,170 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:25,171 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:27,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:27,348 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:27,348 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:29,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:29,555 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:29,555 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:31,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:31,783 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:31,783 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:33,783 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:33,996 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:33,996 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:35,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:36,165 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:36,165 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:38,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:38,352 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:38,352 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:40,352 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:40,517 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:40,517 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:42,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:42,687 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:42,687 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:44,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:44,846 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:44,846 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:46,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:47,005 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:47,005 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:49,005 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:49,186 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:49,186 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:51,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:51,354 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:51,354 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:53,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:53,517 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:53,519 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:55,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:55,680 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:55,681 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:57,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:34:57,846 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:34:57,848 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:34:59,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:35:00,000 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:35:00,000 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:35:02,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:35:02,165 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:35:02,165 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:35:04,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:35:04,328 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:35:04,328 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:35:06,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:35:06,496 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:35:06,496 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:35:08,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:35:08,664 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:35:08,665 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:35:10,665 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:35:10,861 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:35:10,861 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:35:12,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:35:13,012 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:35:13,012 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 101.417999983. 2021-12-02 08:35:13,013 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:35:13,016 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:35:13,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:35:13,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:35:13,457 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:35:13,459 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:35:13,471 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:35:13,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:35:13,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:35:13,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:35:13,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:35:13,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:35:13,602 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:35:13,763 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:35:13,763 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.75. 2021-12-02 08:35:13,765 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:35:13,766 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:35:13,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:35:14,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:35:14,257 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:35:14,257 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:35:14,280 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:35:14,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:35:14,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:35:14,336 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:35:14,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:35:14,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:35:14,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:35:14,599 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:35:14,599 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.834999799728. 2021-12-02 08:35:14,601 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:35:14,602 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:35:14,719 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:35:14,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:35:15,076 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:35:15,076 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:35:15,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:35:15,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:35:15,147 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:35:15,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:35:15,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:35:15,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:35:15,227 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:35:15,454 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:35:15,456 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.856000185013. ############################################################ # # RAM FootPrint: 5546.41015625 # ############################################################ 2021-12-02 08:35:15,486 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-02 08:35:15,486 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 86 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 2021-12-02 08:36:41,490 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-02 08:36:41,490 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-02 08:36:46,493 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:36:46,742 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:36:47,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:36:47,457 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:36:47,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:36:47,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:36:47,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:36:47,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:36:47,553 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:36:47,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:36:47,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:36:47,608 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:36:47,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:36:47,676 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['34116 memcached\n'], []) 2021-12-02 08:36:47,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-02 08:36:47,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-02 08:36:47,808 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['34116 memcached\n'], []) 2021-12-02 08:36:47,809 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:36:47,811 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:36:47,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:36:48,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:36:48,338 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:36:48,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:36:48,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:36:48,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:36:48,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:36:48,431 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:36:48,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:36:48,471 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:36:48,505 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:36:48,505 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:36:48,588 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['109793 memcached\n'], []) 2021-12-02 08:36:48,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-02 08:36:48,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-02 08:36:48,750 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['109793 memcached\n'], []) 2021-12-02 08:36:48,750 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:36:48,755 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:36:48,862 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:36:49,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:36:49,246 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:36:49,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:36:49,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:36:49,285 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:36:49,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:36:49,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:36:49,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:36:49,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:36:49,385 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:36:49,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:36:49,457 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['233373 memcached\n'], []) 2021-12-02 08:36:49,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-02 08:36:49,535 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-02 08:36:49,605 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['233373 memcached\n'], []) 2021-12-02 08:36:49,605 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:36:49,608 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:36:49,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:36:49,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:36:50,029 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:36:50,030 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:36:50,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:36:50,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:36:50,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:36:50,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:36:50,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:36:50,178 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:36:50,206 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 08:36:50,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:36:50,295 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['19133 memcached\n'], []) 2021-12-02 08:36:50,295 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 08:36:50,349 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 08:36:50,392 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['19133 memcached\n'], []) 2021-12-02 08:36:50,392 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:36:50,395 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:36:50,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:36:50,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:36:50,846 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:36:50,846 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:36:50,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:36:50,882 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:36:50,903 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:36:50,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:36:50,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:36:50,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:36:50,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:36:50,977 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 08:36:51,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:36:51,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:51,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:51,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:36:51,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:36:51,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:51,265 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:51,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:36:51,357 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:36:51,387 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:51,438 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:51,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:36:51,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:36:51,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:51,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:51,635 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:36:51,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:36:51,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:51,714 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:51,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:36:51,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:36:51,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:51,826 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:51,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:36:51,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:36:51,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:36:51,934 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 08:36:51,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:36:51,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:51,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:36:51,977 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 08:36:51,977 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:36:51,980 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:36:52,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:36:52,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:36:52,460 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:36:52,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:36:52,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:36:52,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:36:52,522 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:36:52,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:36:52,558 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:36:52,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:36:52,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:36:52,601 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 08:36:52,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:36:52,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:52,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:52,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:36:52,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:36:52,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:52,823 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:52,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:36:52,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:36:52,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:52,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:52,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:36:53,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:36:53,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:53,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:53,122 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:36:53,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:36:53,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:36:53,194 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 08:36:53,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:36:53,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:53,232 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:36:53,249 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 08:36:53,249 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:36:53,253 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:36:53,382 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:36:53,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:36:53,756 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:36:53,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:36:53,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:36:53,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:36:53,826 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:36:53,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:36:53,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:36:53,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:36:53,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:36:53,931 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 08:36:53,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:36:53,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:54,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:54,091 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:36:54,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:36:54,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:54,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:54,272 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:36:54,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:36:54,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:54,395 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:54,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:36:54,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:36:54,522 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:54,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:54,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:36:54,653 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:36:54,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:54,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:54,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:36:54,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:36:54,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:54,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:54,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:36:54,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:36:54,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:54,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:54,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:36:54,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 08:36:54,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:54,989 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:55,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:36:55,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 08:36:55,040 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:55,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:55,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:36:55,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 08:36:55,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:55,127 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:55,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-02 08:36:55,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-02 08:36:55,170 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:36:55,181 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 08:36:55,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:36:55,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:55,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:36:55,227 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 08:36:55,227 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:36:55,230 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:36:55,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:36:55,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:36:55,701 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:36:55,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:36:55,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:36:55,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:36:55,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:36:55,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:36:55,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:36:55,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:36:55,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 08:36:55,855 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 08:36:55,887 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 08:36:55,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:55,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:56,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:36:56,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 08:36:56,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:56,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:56,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:36:56,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 08:36:56,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:56,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:56,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:36:56,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 08:36:56,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:56,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:56,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:36:56,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 08:36:56,505 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:56,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:56,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:36:56,579 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 08:36:56,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:56,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 08:36:56,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:36:56,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 08:36:56,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 08:36:56,717 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 08:36:56,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 08:36:56,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 08:36:56,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 08:36:56,766 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 08:36:56,766 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:36:56,766 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:36:56,769 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 08:36:56,885 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:36:57,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 08:36:57,368 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:36:57,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 08:36:57,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 08:36:57,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 08:36:57,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 08:36:57,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 08:36:57,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 08:36:57,469 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 08:36:57,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:36:57,634 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:36:57,641 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:36:57,641 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:36:59,642 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:36:59,813 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:36:59,819 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:36:59,819 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:01,821 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:01,999 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:37:02,003 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:02,003 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:04,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:04,181 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:37:04,186 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:04,186 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:06,186 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:06,355 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:37:06,358 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:06,358 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:08,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:08,546 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:37:08,549 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:08,549 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:10,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:10,727 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:37:10,734 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:10,736 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:12,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:12,910 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:37:12,914 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:12,914 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:14,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:15,069 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:37:15,075 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:15,075 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:17,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:17,259 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4107] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-02 08:37:17,263 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:17,263 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:19,265 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:19,414 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:19,414 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:21,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:21,561 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:21,561 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:23,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:23,720 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:23,720 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:25,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:25,868 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:25,868 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:27,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:28,016 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:28,016 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:30,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:30,164 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:30,165 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:32,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:32,309 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:32,309 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:34,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:34,540 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:34,540 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:36,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:36,743 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:36,743 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:38,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:38,953 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:38,953 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:40,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:41,168 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:41,170 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:43,170 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:43,361 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:43,361 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:45,362 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:45,576 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:45,576 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:47,576 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:47,766 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:47,766 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:49,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:49,997 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:49,997 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:51,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:52,197 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:52,197 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:54,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:54,392 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:54,394 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:56,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:56,582 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:56,582 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:37:58,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:37:58,796 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:37:58,796 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:00,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:01,006 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:01,006 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:03,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:03,213 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:03,214 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:05,214 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:05,428 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:05,428 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:07,428 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:07,635 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:07,635 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:09,635 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:09,852 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:09,852 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:11,854 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:12,010 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:12,010 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:14,012 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:14,204 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:14,204 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:16,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:16,365 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:16,365 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:18,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:18,540 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:18,540 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:20,540 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:20,713 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:20,713 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:22,714 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:22,887 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:22,888 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:24,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:25,072 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:25,072 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:27,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:27,242 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:27,242 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:29,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:29,430 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:29,430 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:31,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:31,611 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:31,611 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:33,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:33,798 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:33,798 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:35,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:35,979 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:35,979 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:37,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:38,161 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:38,161 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:40,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:40,338 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:40,339 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:42,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:42,515 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4108 2021-12-02 08:38:42,515 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-02 08:38:44,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:44,723 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 08:38:44,723 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 107.95600009. 2021-12-02 08:38:44,723 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:38:44,726 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 08:38:44,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:38:52,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 08:38:52,229 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:38:52,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 08:38:52,247 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 08:38:52,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 08:38:52,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 08:38:52,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 08:38:52,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 08:38:52,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 08:38:52,469 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:52,657 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 08:38:52,658 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 7.93499994278. 2021-12-02 08:38:52,660 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:38:52,661 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 08:38:52,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:38:53,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 08:38:53,180 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:38:53,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 08:38:53,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 08:38:53,219 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 08:38:53,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 08:38:53,265 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 08:38:53,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 08:38:53,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 08:38:53,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:53,480 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 08:38:53,480 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.819999933243. 2021-12-02 08:38:53,482 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4091] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-02 08:38:53,483 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 08:38:53,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 08:38:53,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 08:38:53,900 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 08:38:53,900 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 08:38:53,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 08:38:53,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 08:38:53,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 08:38:53,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 08:38:53,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 08:38:54,012 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 08:38:54,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-02 08:38:54,226 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 08:38:54,226 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.744999885559. 2021-12-02 08:38:54,230 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L835 2021-12-02 08:38:54,230 | test | DEBUG | Thread-1 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None ############################################################ # # Step 6: Rebalance Out with Loading of docs # ############################################################ KV nodes in cluster: ['172.23.110.65', '172.23.110.66', '172.23.110.68'] CBAS nodes in cluster: [] INDEX nodes in cluster: [] FTS nodes in cluster: [] QUERY nodes in cluster: [] EVENTING nodes in cluster: [] AVAILABLE nodes for cluster: ['172.23.110.67'] Servers coming in : [] with services: ['kv'] Servers going out : ['172.23.110.65'] NEW AVAILABLE nodes for cluster: ['172.23.110.67', '172.23.110.65'] 2021-12-02 08:43:54,272 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.65]_1638463434.23 2021-12-02 08:43:54,272 | infra | INFO | pool-3-thread-25 | [task:start_task:89] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.65]_1638463434.23' started 2021-12-02 08:43:54,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L490 2021-12-02 08:43:54,283 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep for 60s for rebalance to start 2021-12-02 08:43:54,309 | test | DEBUG | pool-3-thread-25 | [task:start_rebalance:355] Removing node 172.23.110.65:8091 from cluster 2021-12-02 08:43:54,309 | test | DEBUG | pool-3-thread-25 | [rest_client:rebalance:1431] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.65', 'user': 'Administrator'} 2021-12-02 08:43:54,367 | test | DEBUG | pool-3-thread-25 | [rest_client:rebalance:1436] Rebalance operation started 2021-12-02 08:43:54,371 | test | INFO | pool-3-thread-25 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1817-enterprise | 9.60945027019 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1817-enterprise | 12.5471182546 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1817-enterprise | 10.4102715053 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1817-enterprise | 12.5346163529 | --- OUT ---> | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-02 08:43:54,382 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 08:43:54,382 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 08:43:54,398 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:43:54,401 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:43:54,401 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:44:04,417 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 08:44:04,417 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 08:44:04,434 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:44:04,437 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:44:04,437 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:44:14,451 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 08:44:14,451 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 08:44:14,467 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:44:14,470 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:44:14,470 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:44:24,487 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 08:44:24,487 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 08:44:24,503 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:44:24,506 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:44:24,506 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:44:34,533 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 08:44:34,535 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 08:44:34,552 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:44:34,555 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:44:34,555 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:44:44,568 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 08:44:44,568 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 08:44:44,582 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:44:44,585 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:44:44,585 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:44:54,295 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L765 2021-12-02 08:44:54,295 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 30 seconds. Reason: Let the rebalance begin! 2021-12-02 08:44:54,599 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 08:44:54,599 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 08:44:54,615 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:44:54,618 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:44:54,618 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:45:04,631 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.19 % 2021-12-02 08:45:04,631 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 0.19 2021-12-02 08:45:04,647 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:45:04,648 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:45:04,648 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:45:14,661 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.17 % 2021-12-02 08:45:14,661 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 1.17 2021-12-02 08:45:14,677 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:45:14,678 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:45:14,678 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:45:24,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.17 % 2021-12-02 08:45:24,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:45:24,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:45:24,690 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.17 % 2021-12-02 08:45:24,690 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 1.17 2021-12-02 08:45:24,709 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:45:24,711 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:45:24,711 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:45:34,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.17 % 2021-12-02 08:45:34,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:45:34,364 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:45:34,723 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.17 % 2021-12-02 08:45:34,723 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 1.17 2021-12-02 08:45:34,746 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:45:34,750 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:45:34,750 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:45:44,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.17 % 2021-12-02 08:45:44,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:45:44,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:45:44,763 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.17 % 2021-12-02 08:45:44,763 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 1.17 2021-12-02 08:45:44,788 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:45:44,789 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:45:44,789 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:45:54,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.17 % 2021-12-02 08:45:54,418 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:45:54,418 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:45:54,805 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.17 % 2021-12-02 08:45:54,805 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 1.17 2021-12-02 08:45:54,822 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:45:54,825 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:45:54,825 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:46:04,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.37 % 2021-12-02 08:46:04,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:46:04,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:46:04,838 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.37 % 2021-12-02 08:46:04,838 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 1.37 2021-12-02 08:46:04,855 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:46:04,857 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:46:04,857 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:46:14,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 08:46:14,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:46:14,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:46:14,868 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.56 % 2021-12-02 08:46:14,869 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 1.56 2021-12-02 08:46:14,885 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:46:14,887 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:46:14,888 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:46:24,497 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.76 % 2021-12-02 08:46:24,507 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:46:24,507 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:46:24,898 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.76 % 2021-12-02 08:46:24,898 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 1.76 2021-12-02 08:46:24,921 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:46:24,923 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:46:24,923 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:46:34,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.95 % 2021-12-02 08:46:34,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:46:34,533 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:46:34,936 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 1.95 % 2021-12-02 08:46:34,936 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 1.95 2021-12-02 08:46:34,954 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:46:34,957 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:46:34,957 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:46:44,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 08:46:44,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:46:44,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:46:44,970 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 08:46:44,970 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 08:46:44,987 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:46:44,990 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:46:44,990 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:46:54,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 08:46:54,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:46:54,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:46:55,005 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 08:46:55,005 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 08:46:55,028 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:46:55,029 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:46:55,029 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:47:04,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 08:47:04,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:47:04,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:47:05,043 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.34 % 2021-12-02 08:47:05,043 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 2.34 2021-12-02 08:47:05,059 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:47:05,062 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:47:05,062 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:47:14,622 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.54 % 2021-12-02 08:47:14,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:47:14,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:47:15,075 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.54 % 2021-12-02 08:47:15,075 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 2.54 2021-12-02 08:47:15,098 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:47:15,099 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:47:15,099 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:47:24,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.73 % 2021-12-02 08:47:24,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:47:24,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:47:25,114 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 2.73 % 2021-12-02 08:47:25,115 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 2.73 2021-12-02 08:47:25,130 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:47:25,131 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:47:25,131 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:47:34,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.12 % 2021-12-02 08:47:34,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:47:34,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:47:35,145 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.12 % 2021-12-02 08:47:35,147 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 3.12 2021-12-02 08:47:35,165 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:47:35,168 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:47:35,168 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:47:44,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.12 % 2021-12-02 08:47:44,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:47:44,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:47:45,183 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.12 % 2021-12-02 08:47:45,183 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 3.12 2021-12-02 08:47:45,203 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:47:45,206 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:47:45,206 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:47:54,714 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.12 % 2021-12-02 08:47:54,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:47:54,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:47:55,217 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.12 % 2021-12-02 08:47:55,217 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 3.12 2021-12-02 08:47:55,240 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:47:55,243 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:47:55,243 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:48:04,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.52 % 2021-12-02 08:48:04,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:48:04,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:48:05,256 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.52 % 2021-12-02 08:48:05,256 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 3.52 2021-12-02 08:48:05,273 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:48:05,276 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:48:05,276 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:48:14,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.71 % 2021-12-02 08:48:14,765 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:48:14,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:48:15,289 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.71 % 2021-12-02 08:48:15,289 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 3.71 2021-12-02 08:48:15,305 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:48:15,308 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:48:15,308 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:48:24,778 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.90 % 2021-12-02 08:48:24,785 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:48:24,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:48:25,319 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.90 % 2021-12-02 08:48:25,321 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 3.9 2021-12-02 08:48:25,342 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:48:25,345 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:48:25,345 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:48:34,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 3.90 % 2021-12-02 08:48:34,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:48:34,806 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:48:35,359 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.10 % 2021-12-02 08:48:35,359 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 4.1 2021-12-02 08:48:35,381 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:48:35,384 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:48:35,384 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:48:44,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 08:48:44,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:48:44,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:48:45,395 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 08:48:45,395 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 4.3 2021-12-02 08:48:45,431 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:48:45,434 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:48:45,434 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:48:54,836 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 08:48:54,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:48:54,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:48:55,447 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 08:48:55,447 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 4.3 2021-12-02 08:48:55,464 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:48:55,467 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:48:55,467 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:49:04,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 08:49:04,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:49:04,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:49:05,484 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 08:49:05,486 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 4.3 2021-12-02 08:49:05,509 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:49:05,512 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:49:05,512 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:49:14,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 08:49:14,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:49:14,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:49:15,525 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.30 % 2021-12-02 08:49:15,525 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 4.3 2021-12-02 08:49:15,543 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:49:15,546 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:49:15,546 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:49:24,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.49 % 2021-12-02 08:49:24,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:49:24,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:49:25,559 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.69 % 2021-12-02 08:49:25,559 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 4.69 2021-12-02 08:49:25,584 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:49:25,586 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:49:25,586 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:49:34,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.88 % 2021-12-02 08:49:34,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:49:34,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:49:35,602 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 4.88 % 2021-12-02 08:49:35,602 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 4.88 2021-12-02 08:49:35,621 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:49:35,624 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:49:35,624 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:49:44,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 08:49:44,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:49:44,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:49:45,637 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 08:49:45,638 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 08:49:45,654 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:49:45,657 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:49:45,657 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:49:54,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 08:49:54,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:49:54,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:49:55,671 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 08:49:55,671 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 08:49:55,688 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:49:55,690 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:49:55,690 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:50:04,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 08:50:04,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:50:04,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:50:05,707 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 08:50:05,707 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 08:50:05,724 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:50:05,726 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:50:05,727 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:50:15,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 08:50:15,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:50:15,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:50:15,742 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 08:50:15,742 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 08:50:15,757 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:50:15,759 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:50:15,759 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:50:25,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 08:50:25,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:50:25,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:50:25,776 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.47 % 2021-12-02 08:50:25,776 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 5.47 2021-12-02 08:50:25,795 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:50:25,796 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:50:25,796 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:50:35,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 5.86 % 2021-12-02 08:50:35,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:50:35,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:50:35,808 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.05 % 2021-12-02 08:50:35,809 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 6.05 2021-12-02 08:50:35,826 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:50:35,829 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:50:35,829 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:50:45,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 08:50:45,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:50:45,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:50:45,841 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 08:50:45,842 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 6.25 2021-12-02 08:50:45,861 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:50:45,865 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:50:45,865 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:50:55,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 08:50:55,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:50:55,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:50:55,878 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.25 % 2021-12-02 08:50:55,878 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 6.25 2021-12-02 08:50:55,898 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:50:55,901 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:50:55,901 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:51:05,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.45 % 2021-12-02 08:51:05,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:51:05,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:51:05,913 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.45 % 2021-12-02 08:51:05,914 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 6.45 2021-12-02 08:51:05,928 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:51:05,930 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:51:05,930 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:51:15,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.64 % 2021-12-02 08:51:15,124 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:51:15,124 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:51:15,947 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.64 % 2021-12-02 08:51:15,948 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 6.64 2021-12-02 08:51:15,971 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:51:15,974 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:51:15,974 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:51:25,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.64 % 2021-12-02 08:51:25,142 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:51:25,142 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:51:25,990 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.64 % 2021-12-02 08:51:25,990 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 6.64 2021-12-02 08:51:26,012 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:51:26,013 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:51:26,013 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:51:35,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.83 % 2021-12-02 08:51:35,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:51:35,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:51:36,030 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.83 % 2021-12-02 08:51:36,030 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 6.83 2021-12-02 08:51:36,049 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:51:36,052 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:51:36,052 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:51:45,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.83 % 2021-12-02 08:51:45,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:51:45,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:51:46,063 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 6.83 % 2021-12-02 08:51:46,065 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 6.83 2021-12-02 08:51:46,084 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:51:46,085 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:51:46,085 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:51:55,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 08:51:55,196 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:51:55,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:51:56,098 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 08:51:56,099 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 7.03 2021-12-02 08:51:56,119 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:51:56,121 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:51:56,121 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:52:05,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 08:52:05,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:52:05,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:52:06,137 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.03 % 2021-12-02 08:52:06,137 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 7.03 2021-12-02 08:52:06,158 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:52:06,161 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:52:06,161 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:52:15,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 08:52:15,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:52:15,234 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:52:16,180 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 08:52:16,180 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 7.81 2021-12-02 08:52:16,210 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:52:16,213 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:52:16,213 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:52:25,247 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 08:52:25,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:52:25,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:52:26,223 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 08:52:26,224 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 7.81 2021-12-02 08:52:26,240 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:52:26,242 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:52:26,242 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:52:35,267 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 08:52:35,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:52:35,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:52:36,255 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 7.81 % 2021-12-02 08:52:36,256 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 7.81 2021-12-02 08:52:36,270 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:52:36,273 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:52:36,273 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:52:45,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 08:52:45,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:52:45,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:52:46,289 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 08:52:46,290 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 8.01 2021-12-02 08:52:46,313 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:52:46,315 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:52:46,315 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:52:55,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 08:52:55,318 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:52:55,318 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:52:56,328 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.01 % 2021-12-02 08:52:56,328 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 8.01 2021-12-02 08:52:56,348 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:52:56,351 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:52:56,351 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:53:05,334 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.40 % 2021-12-02 08:53:05,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:53:05,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:53:06,365 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.40 % 2021-12-02 08:53:06,365 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 8.4 2021-12-02 08:53:06,388 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:53:06,391 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:53:06,391 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:53:15,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.40 % 2021-12-02 08:53:15,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:53:15,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:53:16,407 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.40 % 2021-12-02 08:53:16,407 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 8.4 2021-12-02 08:53:16,428 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:53:16,431 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:53:16,431 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:53:25,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.59 % 2021-12-02 08:53:25,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:53:25,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:53:26,444 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.59 % 2021-12-02 08:53:26,444 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 8.59 2021-12-02 08:53:26,460 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:53:26,463 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:53:26,463 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:53:35,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.59 % 2021-12-02 08:53:35,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:53:35,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:53:36,476 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.59 % 2021-12-02 08:53:36,477 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 8.59 2021-12-02 08:53:36,496 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:53:36,497 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:53:36,497 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:53:45,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 08:53:45,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:53:45,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:53:46,509 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 8.79 % 2021-12-02 08:53:46,510 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 8.79 2021-12-02 08:53:46,532 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:53:46,536 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:53:46,536 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:53:55,469 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.18 % 2021-12-02 08:53:55,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:53:55,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:53:56,552 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.18 % 2021-12-02 08:53:56,552 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 9.18 2021-12-02 08:53:56,569 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:53:56,571 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:53:56,572 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:54:05,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.37 % 2021-12-02 08:54:05,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:54:05,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:54:06,586 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.37 % 2021-12-02 08:54:06,586 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 9.37 2021-12-02 08:54:06,602 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:54:06,607 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:54:06,607 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:54:15,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 08:54:15,525 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:54:15,525 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:54:16,621 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 08:54:16,621 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 9.57 2021-12-02 08:54:16,640 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:54:16,642 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:54:16,642 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:54:25,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 08:54:25,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:54:25,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:54:26,655 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.57 % 2021-12-02 08:54:26,657 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 9.57 2021-12-02 08:54:26,674 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:54:26,677 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:54:26,677 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:54:35,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.76 % 2021-12-02 08:54:35,573 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:54:35,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:54:36,690 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.76 % 2021-12-02 08:54:36,691 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 9.76 2021-12-02 08:54:36,709 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:54:36,711 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:54:36,711 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:54:45,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.96 % 2021-12-02 08:54:45,599 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:54:45,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:54:46,723 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 9.96 % 2021-12-02 08:54:46,723 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 9.96 2021-12-02 08:54:46,740 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:54:46,743 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:54:46,743 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:54:55,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 08:54:55,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:54:55,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:54:56,756 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 08:54:56,756 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 10.15 2021-12-02 08:54:56,773 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:54:56,776 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:54:56,778 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:55:05,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.15 % 2021-12-02 08:55:05,650 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:55:05,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:55:06,790 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.35 % 2021-12-02 08:55:06,792 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 10.35 2021-12-02 08:55:06,808 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:55:06,809 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:55:06,809 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:55:15,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 08:55:15,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:55:15,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:55:16,825 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 08:55:16,825 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 10.74 2021-12-02 08:55:16,842 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:55:16,844 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:55:16,845 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:55:25,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 08:55:25,696 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:55:25,696 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:55:26,855 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 08:55:26,855 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 10.74 2021-12-02 08:55:26,871 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:55:26,874 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:55:26,874 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:55:35,709 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 08:55:35,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:55:35,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:55:36,888 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 08:55:36,888 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 10.74 2021-12-02 08:55:36,905 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:55:36,907 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:55:36,907 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:55:45,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 08:55:45,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:55:45,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:55:46,921 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.74 % 2021-12-02 08:55:46,923 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 10.74 2021-12-02 08:55:46,940 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:55:46,943 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:55:46,944 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:55:55,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.93 % 2021-12-02 08:55:55,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:55:55,783 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:55:56,957 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.93 % 2021-12-02 08:55:56,957 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 10.93 2021-12-02 08:55:56,977 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:55:56,980 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:55:56,980 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:56:05,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.93 % 2021-12-02 08:56:05,808 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:56:05,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:56:06,994 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 10.93 % 2021-12-02 08:56:06,996 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 10.93 2021-12-02 08:56:07,012 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:56:07,013 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:56:07,013 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:56:15,821 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.13 % 2021-12-02 08:56:15,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:56:15,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:56:17,026 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.32 % 2021-12-02 08:56:17,026 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 11.32 2021-12-02 08:56:17,042 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:56:17,045 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:56:17,046 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:56:25,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.52 % 2021-12-02 08:56:25,857 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:56:25,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:56:27,062 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.72 % 2021-12-02 08:56:27,063 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 11.72 2021-12-02 08:56:27,084 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:56:27,085 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:56:27,085 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:56:35,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.91 % 2021-12-02 08:56:35,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:56:35,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:56:37,096 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.91 % 2021-12-02 08:56:37,096 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 11.91 2021-12-02 08:56:37,112 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:56:37,115 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:56:37,115 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:56:45,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.91 % 2021-12-02 08:56:45,892 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:56:45,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:56:47,128 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 11.91 % 2021-12-02 08:56:47,128 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 11.91 2021-12-02 08:56:47,147 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:56:47,150 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:56:47,151 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:56:55,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.11 % 2021-12-02 08:56:55,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:56:55,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:56:57,165 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.11 % 2021-12-02 08:56:57,165 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 12.11 2021-12-02 08:56:57,187 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:56:57,188 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:56:57,188 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:57:05,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.11 % 2021-12-02 08:57:05,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:57:05,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:57:07,203 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.11 % 2021-12-02 08:57:07,204 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 12.11 2021-12-02 08:57:07,223 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:57:07,226 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:57:07,226 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:57:15,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.11 % 2021-12-02 08:57:15,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:57:15,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:57:17,239 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.11 % 2021-12-02 08:57:17,239 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 12.11 2021-12-02 08:57:17,263 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:57:17,266 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:57:17,266 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:57:25,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.11 % 2021-12-02 08:57:25,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:57:25,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:57:27,279 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.30 % 2021-12-02 08:57:27,279 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 12.3 2021-12-02 08:57:27,299 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:57:27,301 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:57:27,301 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:57:35,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.69 % 2021-12-02 08:57:35,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:57:35,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:57:37,315 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.69 % 2021-12-02 08:57:37,315 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 12.69 2021-12-02 08:57:37,329 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:57:37,332 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:57:37,332 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:57:46,007 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.69 % 2021-12-02 08:57:46,013 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:57:46,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:57:47,344 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.69 % 2021-12-02 08:57:47,344 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 12.69 2021-12-02 08:57:47,358 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:57:47,361 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:57:47,361 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:57:56,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 12.89 % 2021-12-02 08:57:56,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:57:56,033 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:57:57,377 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.08 % 2021-12-02 08:57:57,378 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 13.08 2021-12-02 08:57:57,401 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:57:57,404 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:57:57,404 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:58:06,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.08 % 2021-12-02 08:58:06,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:58:06,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:58:07,417 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.08 % 2021-12-02 08:58:07,418 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 13.08 2021-12-02 08:58:07,441 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:58:07,444 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:58:07,446 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:58:16,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.08 % 2021-12-02 08:58:16,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:58:16,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:58:17,457 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.08 % 2021-12-02 08:58:17,457 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 13.08 2021-12-02 08:58:17,476 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:58:17,480 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:58:17,480 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:58:26,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 08:58:26,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:58:26,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:58:27,492 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.28 % 2021-12-02 08:58:27,492 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 13.28 2021-12-02 08:58:27,512 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:58:27,513 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:58:27,515 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:58:36,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.67 % 2021-12-02 08:58:36,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:58:36,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:58:37,529 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.67 % 2021-12-02 08:58:37,529 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 13.67 2021-12-02 08:58:37,546 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:58:37,549 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:58:37,549 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:58:46,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.86 % 2021-12-02 08:58:46,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:58:46,132 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:58:47,565 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.86 % 2021-12-02 08:58:47,565 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 13.86 2021-12-02 08:58:47,582 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:58:47,585 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:58:47,585 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:58:56,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.86 % 2021-12-02 08:58:56,153 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:58:56,153 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:58:57,595 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.86 % 2021-12-02 08:58:57,595 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 13.86 2021-12-02 08:58:57,614 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:58:57,618 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:58:57,618 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:59:06,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.86 % 2021-12-02 08:59:06,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:59:06,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:59:07,630 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.86 % 2021-12-02 08:59:07,631 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 13.86 2021-12-02 08:59:07,647 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:59:07,650 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:59:07,650 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:59:16,186 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 13.86 % 2021-12-02 08:59:16,191 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:59:16,191 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:59:17,665 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.06 % 2021-12-02 08:59:17,665 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 14.06 2021-12-02 08:59:17,687 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:59:17,690 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:59:17,690 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:59:26,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.26 % 2021-12-02 08:59:26,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:59:26,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:59:27,703 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.26 % 2021-12-02 08:59:27,703 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 14.26 2021-12-02 08:59:27,719 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:59:27,721 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:59:27,721 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:59:36,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.65 % 2021-12-02 08:59:36,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:59:36,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:59:37,736 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.65 % 2021-12-02 08:59:37,736 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 14.65 2021-12-02 08:59:37,755 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:59:37,757 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:59:37,759 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:59:46,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.84 % 2021-12-02 08:59:46,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:59:46,255 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:59:47,772 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.84 % 2021-12-02 08:59:47,773 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 14.84 2021-12-02 08:59:47,790 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:59:47,793 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:59:47,793 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:59:56,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 14.84 % 2021-12-02 08:59:56,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 08:59:56,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 08:59:57,805 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.04 % 2021-12-02 08:59:57,806 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 15.04 2021-12-02 08:59:57,823 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 08:59:57,826 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 08:59:57,828 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:00:06,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.04 % 2021-12-02 09:00:06,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:00:06,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:00:07,842 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.04 % 2021-12-02 09:00:07,842 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 15.04 2021-12-02 09:00:07,859 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:00:07,862 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:00:07,864 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:00:16,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.04 % 2021-12-02 09:00:16,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:00:16,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:00:17,875 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.04 % 2021-12-02 09:00:17,877 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 15.04 2021-12-02 09:00:17,891 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:00:17,894 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:00:17,894 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:00:26,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.04 % 2021-12-02 09:00:26,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:00:26,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:00:27,911 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.04 % 2021-12-02 09:00:27,911 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 15.04 2021-12-02 09:00:27,928 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:00:27,930 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:00:27,931 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:00:36,346 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.23 % 2021-12-02 09:00:36,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:00:36,352 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:00:37,944 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.23 % 2021-12-02 09:00:37,944 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 15.23 2021-12-02 09:00:37,960 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:00:37,963 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:00:37,963 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:00:46,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.62 % 2021-12-02 09:00:46,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:00:46,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:00:47,977 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.62 % 2021-12-02 09:00:47,977 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 15.62 2021-12-02 09:00:48,013 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:00:48,016 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:00:48,016 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:00:56,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.82 % 2021-12-02 09:00:56,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:00:56,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:00:58,028 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 15.82 % 2021-12-02 09:00:58,029 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 15.82 2021-12-02 09:00:58,045 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:00:58,046 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:00:58,046 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:01:06,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.01 % 2021-12-02 09:01:06,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:01:06,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:01:08,062 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 09:01:08,062 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 16.21 2021-12-02 09:01:08,078 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:01:08,082 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:01:08,082 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:01:16,426 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 09:01:16,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:01:16,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:01:18,095 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 09:01:18,095 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 16.21 2021-12-02 09:01:18,119 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:01:18,121 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:01:18,121 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:01:26,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 09:01:26,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:01:26,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:01:28,134 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 09:01:28,135 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 16.21 2021-12-02 09:01:28,157 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:01:28,160 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:01:28,160 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:01:36,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 09:01:36,482 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:01:36,482 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:01:38,173 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.21 % 2021-12-02 09:01:38,173 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 16.21 2021-12-02 09:01:38,193 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:01:38,194 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:01:38,194 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:01:46,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.40 % 2021-12-02 09:01:46,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:01:46,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:01:48,209 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.40 % 2021-12-02 09:01:48,209 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 16.4 2021-12-02 09:01:48,226 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:01:48,227 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:01:48,227 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:01:56,517 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.79 % 2021-12-02 09:01:56,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:01:56,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:01:58,243 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.79 % 2021-12-02 09:01:58,243 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 16.79 2021-12-02 09:01:58,270 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:01:58,273 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:01:58,273 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:02:06,543 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.99 % 2021-12-02 09:02:06,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:02:06,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:02:08,288 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.99 % 2021-12-02 09:02:08,289 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 16.99 2021-12-02 09:02:08,311 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:02:08,312 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:02:08,313 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:02:16,566 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.99 % 2021-12-02 09:02:16,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:02:16,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:02:18,326 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.99 % 2021-12-02 09:02:18,328 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 16.99 2021-12-02 09:02:18,345 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:02:18,348 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:02:18,348 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:02:26,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.99 % 2021-12-02 09:02:26,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:02:26,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:02:28,362 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.99 % 2021-12-02 09:02:28,362 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 16.99 2021-12-02 09:02:28,388 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:02:28,391 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:02:28,391 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:02:36,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.99 % 2021-12-02 09:02:36,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:02:36,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:02:38,407 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.99 % 2021-12-02 09:02:38,407 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 16.99 2021-12-02 09:02:38,434 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:02:38,436 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:02:38,436 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:02:46,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 16.99 % 2021-12-02 09:02:46,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:02:46,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:02:48,457 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.18 % 2021-12-02 09:02:48,457 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 17.18 2021-12-02 09:02:48,483 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:02:48,486 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:02:48,486 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:02:56,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.77 % 2021-12-02 09:02:56,658 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:02:56,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:02:58,497 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.77 % 2021-12-02 09:02:58,497 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 17.77 2021-12-02 09:02:58,515 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:02:58,517 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:02:58,517 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:03:06,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 09:03:06,681 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:03:06,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:03:08,532 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 17.97 % 2021-12-02 09:03:08,532 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 17.97 2021-12-02 09:03:08,549 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:03:08,552 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:03:08,552 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:03:16,693 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.16 % 2021-12-02 09:03:16,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:03:16,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:03:18,568 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.16 % 2021-12-02 09:03:18,569 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 18.16 2021-12-02 09:03:18,585 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:03:18,586 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:03:18,588 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:03:26,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.16 % 2021-12-02 09:03:26,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:03:26,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:03:28,599 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.16 % 2021-12-02 09:03:28,599 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 18.16 2021-12-02 09:03:28,618 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:03:28,619 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:03:28,619 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:03:36,734 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.16 % 2021-12-02 09:03:36,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:03:36,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:03:38,634 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.16 % 2021-12-02 09:03:38,634 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 18.16 2021-12-02 09:03:38,651 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:03:38,654 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:03:38,654 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:03:46,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.16 % 2021-12-02 09:03:46,765 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:03:46,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:03:48,665 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.16 % 2021-12-02 09:03:48,665 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 18.16 2021-12-02 09:03:48,684 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:03:48,687 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:03:48,687 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:03:56,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.16 % 2021-12-02 09:03:56,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:03:56,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:03:58,700 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.35 % 2021-12-02 09:03:58,700 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 18.35 2021-12-02 09:03:58,719 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:03:58,721 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:03:58,721 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:04:06,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.74 % 2021-12-02 09:04:06,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:04:06,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:04:08,733 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.74 % 2021-12-02 09:04:08,733 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 18.74 2021-12-02 09:04:08,755 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:04:08,759 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:04:08,759 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:04:16,823 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.94 % 2021-12-02 09:04:16,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:04:16,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:04:18,773 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 18.94 % 2021-12-02 09:04:18,773 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 18.94 2021-12-02 09:04:18,790 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:04:18,793 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:04:18,793 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:04:26,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 09:04:26,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:04:26,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:04:28,806 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 09:04:28,808 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 19.14 2021-12-02 09:04:28,823 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:04:28,826 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:04:28,826 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:04:36,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 09:04:36,887 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:04:36,887 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:04:38,839 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.14 % 2021-12-02 09:04:38,839 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 19.14 2021-12-02 09:04:38,858 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:04:38,862 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:04:38,862 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:04:46,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.33 % 2021-12-02 09:04:46,913 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:04:46,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:04:48,874 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.33 % 2021-12-02 09:04:48,874 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 19.33 2021-12-02 09:04:48,890 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:04:48,891 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:04:48,892 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:04:56,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.33 % 2021-12-02 09:04:56,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:04:56,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:04:58,904 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.33 % 2021-12-02 09:04:58,904 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 19.33 2021-12-02 09:04:58,921 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:04:58,924 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:04:58,924 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:05:06,951 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.72 % 2021-12-02 09:05:06,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:05:06,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:05:08,937 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.92 % 2021-12-02 09:05:08,937 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 19.92 2021-12-02 09:05:08,954 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:05:08,957 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:05:08,957 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:05:16,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.92 % 2021-12-02 09:05:16,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:05:16,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:05:18,974 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.92 % 2021-12-02 09:05:18,974 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 19.92 2021-12-02 09:05:18,992 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:05:18,994 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:05:18,994 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:05:26,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.92 % 2021-12-02 09:05:27,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:05:27,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:05:29,012 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 19.92 % 2021-12-02 09:05:29,012 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 19.92 2021-12-02 09:05:29,029 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:05:29,032 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:05:29,032 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:05:37,020 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.31 % 2021-12-02 09:05:37,030 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:05:37,030 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:05:39,046 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.31 % 2021-12-02 09:05:39,046 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 20.31 2021-12-02 09:05:39,063 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:05:39,065 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:05:39,065 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:05:47,032 | test | INFO | MainThread | [rest_client:rebalance_reached:119] Rebalance reached >20.31% in 1222.72000003 seconds 2021-12-02 09:05:47,046 | test | DEBUG | MainThread | [rest_client:diag_eval:1453] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-02 09:05:47,046 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-02 09:05:47,049 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 09:05:47,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 09:05:47,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 09:05:47,401 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 09:05:47,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 09:05:47,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 09:05:47,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 09:05:47,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 09:05:47,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 09:05:47,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 09:05:47,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 09:05:47,528 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-02 09:05:47,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 09:05:47,586 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['20335 memcached\n'], []) 2021-12-02 09:05:47,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-02 09:05:47,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-02 09:05:47,680 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['20335 memcached\n'], []) 2021-12-02 09:05:47,680 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 09:05:47,684 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 09:05:47,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 09:05:48,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 09:05:48,124 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 09:05:48,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 09:05:48,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 09:05:48,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 09:05:48,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 09:05:48,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 09:05:48,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 09:05:48,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 09:05:48,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 09:05:48,256 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 09:05:48,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 09:05:48,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:48,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:48,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:05:48,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:05:48,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:48,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:48,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:05:48,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:05:48,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:48,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:48,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:05:48,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:05:48,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:48,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:48,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:05:48,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:05:48,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:48,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:48,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:05:48,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:05:48,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 09:05:48,786 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 09:05:48,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 09:05:48,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:48,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 09:05:48,835 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 09:05:48,835 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 09:05:48,838 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 09:05:48,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 09:05:49,079 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.31 % 2021-12-02 09:05:49,079 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 20.31 2021-12-02 09:05:49,105 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:05:49,107 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:05:49,107 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:05:49,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 09:05:49,279 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 09:05:49,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 09:05:49,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 09:05:49,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 09:05:49,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 09:05:49,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 09:05:49,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 09:05:49,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 09:05:49,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 09:05:49,428 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 09:05:49,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 09:05:49,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:49,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:49,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:05:49,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:05:49,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:49,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:49,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:05:49,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:05:49,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:49,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:49,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:05:49,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:05:49,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:49,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:49,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:05:49,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:05:49,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:50,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:50,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:05:50,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:05:50,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:50,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:50,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:05:50,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:05:50,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:50,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:50,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 09:05:50,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 09:05:50,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:50,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:50,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 09:05:50,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 09:05:50,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:50,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:50,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 09:05:50,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 09:05:50,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:50,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:50,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-02 09:05:50,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-02 09:05:50,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 09:05:50,559 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 09:05:50,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 09:05:50,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:50,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 09:05:50,602 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 09:05:50,602 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 09:05:50,605 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 09:05:50,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 09:05:51,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 09:05:51,101 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 09:05:51,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 09:05:51,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 09:05:51,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 09:05:51,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 09:05:51,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 09:05:51,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 09:05:51,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 09:05:51,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 09:05:51,267 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 09:05:51,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 09:05:51,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:51,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:51,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:05:51,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:05:51,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:51,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:51,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:05:51,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:05:51,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:51,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:51,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:05:51,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:05:51,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:51,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:51,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:05:51,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:05:51,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:51,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:51,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:05:51,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:05:51,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:51,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:05:51,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:05:51,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:05:51,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 09:05:52,005 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 09:05:52,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 09:05:52,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:05:52,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 09:05:52,053 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 09:05:52,055 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 09:05:52,063 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-02 09:05:52,063 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-02 09:05:59,122 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 20.31 % 2021-12-02 09:05:59,122 | test | INFO | pool-3-thread-25 | [task:check:384] Rebalance - status: running, progress: 20.31 2021-12-02 09:05:59,148 | infra | DEBUG | pool-3-thread-25 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:05:59,150 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:05:59,150 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:06:09,160 | test | ERROR | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1640] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'd7163a66670bafdb09e2aefdf26f186e', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=d791c8049a1d62bb0cdec99af2357e8e', u'status': u'notRunning'} - rebalance failed 2021-12-02 09:06:09,203 | test | INFO | pool-3-thread-25 | [rest_client:print_UI_logs:2789] Latest logs from UI on 172.23.110.65: 2021-12-02 09:06:09,203 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638464759709L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:05:59.709Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1202 08:37:08.710894 20566 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-02 09:06:09,203 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638464759700L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:05:59.700Z', u'text': u"Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {'EXIT',<0.17831.141>,\n {socket_closed,\n {gen_server,call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.110.64'},\n {if_rebalance,<0.12869.139>,\n {wait_dcp_data_move,\n ['ns_1@172.23.110.68'],\n 66}},\n infinity]}}}}}.\nRebalance Operation Id = 9625a7ee5a3fbbddbac7eeccb99d4b48"} 2021-12-02 09:06:09,203 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638464759678L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:05:59.678Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-02 09:06:09,203 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638464759672L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:05:59.672Z', u'text': u"Worker <0.16809.141> (for action {move,\n {66,\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.65'],\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.68'],\n []}}) exited with reason {unexpected_exit,\n {'EXIT',\n <0.17831.141>,\n {socket_closed,\n {gen_server,\n call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.110.64'},\n {if_rebalance,\n <0.12869.139>,\n {wait_dcp_data_move,\n ['ns_1@172.23.110.68'],\n 66}},\n infinity]}}}}"} 2021-12-02 09:06:09,203 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638463434628L, u'shortText': u'message', u'serverTime': u'2021-12-02T08:43:54.628Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-02 09:06:09,204 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638463434506L, u'shortText': u'message', u'serverTime': u'2021-12-02T08:43:54.506Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-02 09:06:09,204 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638463434314L, u'shortText': u'message', u'serverTime': u'2021-12-02T08:43:54.314Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.68',\n 'ns_1@172.23.110.66'], EjectNodes = ['ns_1@172.23.110.65'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 9625a7ee5a3fbbddbac7eeccb99d4b48"} 2021-12-02 09:06:09,204 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.66', u'tstamp': 1638463397568L, u'shortText': u'message', u'serverTime': u'2021-12-02T08:43:17.568Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.66\' in 363 seconds.'} 2021-12-02 09:06:09,204 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.65', u'tstamp': 1638463383290L, u'shortText': u'message', u'serverTime': u'2021-12-02T08:43:03.290Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.65\' in 345 seconds.'} 2021-12-02 09:06:09,204 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638463328734L, u'shortText': u'message', u'serverTime': u'2021-12-02T08:42:08.734Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 291 seconds.'} 2021-12-02 09:06:09,204 | test | ERROR | pool-3-thread-25 | [task:call:296] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'd7163a66670bafdb09e2aefdf26f186e', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=d791c8049a1d62bb0cdec99af2357e8e', u'status': u'notRunning'} - rebalance failed 2021-12-02 09:06:52,081 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.68:8091 2021-12-02 09:06:52,081 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 09:06:52,094 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.68:8091 is running 2021-12-02 09:06:52,104 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-02 09:06:52,105 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 09:06:52,114 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:06:52,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:06:52,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:06:54,134 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:06:54,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:06:54,140 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:06:56,154 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:06:56,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:06:56,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:06:58,174 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:06:58,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:06:58,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:00,190 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:00,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:00,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:02,209 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:02,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:02,216 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:04,227 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:04,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:04,234 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:06,249 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:06,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:06,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:08,269 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:08,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:08,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:10,290 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:10,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:10,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:12,309 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:12,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:12,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:14,331 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:14,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:14,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:16,351 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:16,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:16,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:18,371 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:18,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:18,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:20,391 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:20,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:20,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:22,411 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:22,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:22,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:24,428 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:24,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:24,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:26,450 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:26,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:26,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:28,470 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:28,479 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:28,479 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:30,490 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:30,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:30,499 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:32,512 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:32,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:32,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:34,535 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:34,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:34,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:36,555 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:36,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:36,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:38,575 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:38,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:38,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:40,594 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:40,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:40,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:42,614 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:42,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:42,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:44,648 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:44,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:44,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:46,668 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:46,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:46,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:48,687 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:48,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:48,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:50,706 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:50,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:50,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:52,723 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:52,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:52,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:54,742 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:54,749 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:54,749 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:56,760 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:56,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:56,767 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:07:58,780 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:07:58,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:07:58,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:00,805 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:00,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:00,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:02,831 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:02,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:02,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:04,857 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:04,862 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:04,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:06,878 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:06,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:06,885 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:08,898 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:08,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:08,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:10,920 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:10,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:10,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:12,940 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:12,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:12,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:14,961 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:14,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:14,969 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:16,982 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:16,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:16,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:19,000 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:19,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:19,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:21,020 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:21,028 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:21,028 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:23,043 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:23,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:23,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:25,071 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:25,076 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:25,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:27,092 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:27,099 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:27,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:29,115 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:29,124 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:29,124 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:31,138 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:31,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:31,145 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:33,161 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:33,170 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:33,170 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:35,184 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:35,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:35,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:37,204 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:37,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:37,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:39,223 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:39,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:39,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:41,242 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:41,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:41,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:43,262 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:43,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:43,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:45,285 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:45,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:45,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:47,323 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:47,329 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:47,331 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:49,351 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:49,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:49,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:51,371 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:51,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:51,380 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:53,391 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:53,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:53,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:55,413 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:55,420 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:55,420 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:57,430 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:57,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:57,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:08:59,448 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:08:59,456 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:08:59,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:01,470 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:01,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:01,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:03,490 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:03,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:03,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:05,513 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:05,520 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:05,520 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:07,536 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:07,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:07,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:09,559 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:09,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:09,586 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:11,601 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:11,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:11,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:13,622 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:13,630 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:13,630 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:15,642 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:15,650 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:15,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:17,664 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:17,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:17,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:19,683 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:19,688 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:19,688 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:21,703 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:21,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:21,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:23,723 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:23,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:23,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:25,742 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:25,749 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:25,749 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:27,759 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:27,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:27,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:29,779 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:29,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:29,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:31,799 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:31,808 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:31,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:33,822 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:33,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:33,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:35,845 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:35,854 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:35,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:37,871 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:37,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:37,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:39,895 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:39,903 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:39,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:41,917 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:41,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:41,926 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:43,937 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:43,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:43,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:45,967 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:45,974 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:45,974 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:47,990 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:47,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:47,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:50,016 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:50,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:50,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:52,040 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:52,048 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:52,048 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:54,061 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:54,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:54,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:56,084 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:56,091 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:56,091 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:09:58,105 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:09:58,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:09:58,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:00,128 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:00,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:00,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:02,150 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:02,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:02,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:04,176 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:04,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:04,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:06,200 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:06,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:06,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:08,223 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:08,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:08,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:10,250 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:10,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:10,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:12,269 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:12,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:12,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:14,290 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:14,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:14,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:16,312 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:16,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:16,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:18,334 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:18,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:18,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:20,357 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:20,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:20,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:22,380 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:22,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:22,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:24,403 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:24,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:24,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:26,426 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:26,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:26,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:28,451 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:28,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:28,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:30,476 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:30,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:30,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:32,500 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:32,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:32,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:34,526 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:34,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:34,535 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:36,546 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:36,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:36,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:38,565 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:38,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:38,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:40,588 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:40,596 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:40,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:42,611 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:42,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:42,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:44,631 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:44,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:44,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:46,651 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:46,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:46,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:48,671 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:48,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:48,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:50,690 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:50,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:50,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:52,713 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:52,721 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:52,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:54,736 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:54,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:54,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:56,755 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:56,762 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:56,762 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:10:58,776 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:10:58,783 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:10:58,783 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:00,799 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:00,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:00,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:02,819 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:02,826 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:02,826 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:04,842 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:04,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:04,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:06,864 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:06,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:06,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:08,887 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:08,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:08,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:10,905 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:10,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:10,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:12,927 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:12,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:12,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:14,954 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:14,964 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:14,964 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:16,983 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:16,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:16,990 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:19,005 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:19,012 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:19,012 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:21,026 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:21,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:21,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:23,046 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:23,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:23,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:25,066 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:25,073 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:25,073 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:27,086 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:27,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:27,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:29,108 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:29,114 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:29,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:31,128 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:31,135 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:31,135 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:33,151 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:33,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:33,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:35,173 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:35,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:35,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:37,197 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:37,206 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:37,206 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:39,220 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:39,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:39,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:41,243 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:41,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:41,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:43,266 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:43,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:43,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:45,288 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:45,295 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:45,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:47,315 | test | WARNING | MainThread | [rest_client:is_ns_server_running:68] Server 172.23.110.64:8091 status is warmup 2021-12-02 09:11:47,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L80 2021-12-02 09:11:47,323 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-02 09:11:49,336 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-02 09:11:49,349 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-02 09:11:49,349 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:57] Checking if ns_server is running with timeout 1200secs 2021-12-02 09:11:49,364 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-02 09:11:49,365 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 09:11:49,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 09:11:49,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 09:11:49,729 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 09:11:49,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 09:11:49,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 09:11:49,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 09:11:49,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 09:11:49,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 09:11:49,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 09:11:49,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 09:11:49,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 09:11:49,825 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 09:11:49,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 09:11:49,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:49,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:49,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:11:49,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:11:50,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:50,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:50,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:11:50,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:11:50,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:50,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:50,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:11:50,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:11:50,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:50,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:50,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:11:50,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:11:50,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:50,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:50,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:11:50,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:11:50,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 09:11:50,427 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 09:11:50,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 09:11:50,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:50,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 09:11:50,469 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 09:11:50,469 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 09:11:50,473 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 09:11:50,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 09:11:50,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 09:11:50,907 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 09:11:50,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 09:11:50,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 09:11:50,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 09:11:50,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 09:11:51,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 09:11:51,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 09:11:51,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 09:11:51,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 09:11:51,095 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 09:11:51,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 09:11:51,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:51,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:51,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:11:51,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:11:51,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:51,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:51,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:11:51,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:11:51,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:51,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:51,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:11:51,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:11:51,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:51,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:51,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:11:51,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:11:51,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:51,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:51,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:11:51,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:11:51,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:51,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:51,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:11:51,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:11:51,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:51,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:51,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 09:11:51,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 09:11:51,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:51,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:51,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 09:11:51,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 09:11:52,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:52,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:52,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 09:11:52,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 09:11:52,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:52,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:52,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-02 09:11:52,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-02 09:11:52,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:52,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:52,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-02 09:11:52,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-02 09:11:52,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 09:11:52,375 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 09:11:52,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 09:11:52,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:52,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 09:11:52,417 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 09:11:52,417 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 09:11:52,421 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 09:11:52,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 09:11:52,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 09:11:52,750 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 09:11:52,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 09:11:52,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 09:11:52,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 09:11:52,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 09:11:52,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 09:11:52,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 09:11:52,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 09:11:52,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 09:11:52,848 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 09:11:52,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 09:11:52,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:52,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:52,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:11:52,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:11:53,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:53,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:53,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:11:53,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:11:53,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:53,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:53,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:11:53,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:11:53,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:53,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:53,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:11:53,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:11:53,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:53,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:53,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:11:53,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:11:53,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:53,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:11:53,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:11:53,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:11:53,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 09:11:53,536 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 09:11:53,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 09:11:53,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:11:53,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 09:11:53,588 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 09:11:53,588 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 09:11:53,588 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.110.65]_1638463434.23 2021-12-02 09:11:53,588 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 20 2021-12-02 09:11:53,621 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.65]_1638465113.59 2021-12-02 09:11:53,624 | infra | INFO | pool-3-thread-26 | [task:start_task:89] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.65]_1638465113.59' started 2021-12-02 09:11:53,637 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-02 09:11:53,637 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-02 09:11:53,648 | test | DEBUG | pool-3-thread-26 | [task:start_rebalance:355] Removing node 172.23.110.65:8091 from cluster 2021-12-02 09:11:53,648 | test | DEBUG | pool-3-thread-26 | [rest_client:rebalance:1431] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.65', 'user': 'Administrator'} 2021-12-02 09:11:53,701 | test | DEBUG | pool-3-thread-26 | [rest_client:rebalance:1436] Rebalance operation started 2021-12-02 09:11:53,703 | test | INFO | pool-3-thread-26 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1817-enterprise | 2.87062688811 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1817-enterprise | 1.85638401424 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1817-enterprise | 3.65610859729 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1817-enterprise | 2.74394063844 | --- OUT ---> | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-02 09:11:53,710 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:11:53,711 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:11:53,724 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:11:53,726 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:11:53,727 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:12:03,740 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:12:03,740 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:12:03,757 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:12:03,759 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:12:03,759 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:12:13,773 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:12:13,773 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:12:13,790 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:12:13,792 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:12:13,792 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:12:23,808 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:12:23,808 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:12:23,826 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:12:23,829 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:12:23,829 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:12:33,842 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:12:33,842 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:12:33,858 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:12:33,859 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:12:33,859 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:12:43,875 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:12:43,875 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:12:43,894 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:12:43,895 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:12:43,895 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:12:53,907 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:12:53,908 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:12:53,923 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:12:53,924 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:12:53,924 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:13:03,937 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:13:03,937 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:13:03,953 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:13:03,956 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:13:03,956 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:13:13,967 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:13:13,969 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:13:13,984 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:13:13,987 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:13:13,987 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:13:24,000 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:13:24,000 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:13:24,019 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:13:24,020 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:13:24,022 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:13:34,036 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:13:34,036 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:13:34,056 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:13:34,059 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:13:34,059 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:13:44,072 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:13:44,072 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:13:44,088 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:13:44,091 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:13:44,091 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:13:53,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:13:53,651 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 0.0 2021-12-02 09:13:53,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:13:53,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:13:53,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:13:54,105 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:13:54,105 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:13:54,124 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:13:54,125 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:13:54,125 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:14:03,691 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:14:03,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:14:03,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:14:04,137 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:14:04,138 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:14:04,154 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:14:04,157 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:14:04,157 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:14:13,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:14:13,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:14:13,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:14:14,170 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:14:14,170 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:14:14,190 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:14:14,191 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:14:14,191 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:14:23,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:14:23,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:14:23,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:14:24,203 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:14:24,203 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:14:24,220 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:14:24,220 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:14:24,221 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:14:33,755 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:14:33,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:14:33,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:14:34,233 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:14:34,233 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:14:34,250 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:14:34,252 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:14:34,252 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:14:43,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:14:43,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:14:43,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:14:44,265 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.00 % 2021-12-02 09:14:44,266 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-12-02 09:14:44,286 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:14:44,289 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:14:44,289 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:14:53,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:14:53,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:14:53,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:14:54,301 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:14:54,302 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.25 2021-12-02 09:14:54,322 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:14:54,325 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:14:54,325 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:15:03,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:15:03,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:15:03,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:15:04,344 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:15:04,345 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.25 2021-12-02 09:15:04,365 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:15:04,368 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:15:04,368 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:15:13,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:15:13,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:15:13,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:15:14,380 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:15:14,380 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.25 2021-12-02 09:15:14,398 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:15:14,401 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:15:14,401 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:15:23,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:15:23,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:15:23,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:15:24,413 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:15:24,414 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.25 2021-12-02 09:15:24,431 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:15:24,434 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:15:24,434 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:15:33,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:15:33,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:15:33,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:15:34,448 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:15:34,450 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.25 2021-12-02 09:15:34,467 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:15:34,470 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:15:34,470 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:15:43,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:15:43,887 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:15:43,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:15:44,483 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:15:44,483 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.25 2021-12-02 09:15:44,500 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:15:44,502 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:15:44,503 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:15:53,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:15:53,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:15:53,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:15:54,513 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:15:54,513 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.25 2021-12-02 09:15:54,532 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:15:54,536 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:15:54,536 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:16:03,920 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:16:03,927 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:16:03,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:16:04,549 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:16:04,549 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.25 2021-12-02 09:16:04,568 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:16:04,571 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:16:04,571 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:16:13,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:16:13,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:16:13,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:16:14,581 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:16:14,582 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.25 2021-12-02 09:16:14,602 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:16:14,607 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:16:14,607 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:16:23,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:16:23,966 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:16:23,966 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:16:24,618 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:16:24,618 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.25 2021-12-02 09:16:24,634 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:16:24,638 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:16:24,638 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:16:33,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:16:33,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L109 2021-12-02 09:16:33,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:16:34,653 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1645] Rebalance percentage: 0.25 % 2021-12-02 09:16:34,653 | test | INFO | pool-3-thread-26 | [task:check:384] Rebalance - status: running, progress: 0.25 2021-12-02 09:16:34,671 | infra | DEBUG | pool-3-thread-26 | [task:check:411] Wait before next rebalance progress check 2021-12-02 09:16:34,674 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L412 2021-12-02 09:16:34,674 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-02 09:16:43,996 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1640] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'2bd2c283ef5696ca470581e50b195ddf', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=480565efe39ce4c365be1d6aa68de9da', u'status': u'notRunning'} - rebalance failed 2021-12-02 09:16:44,026 | test | INFO | MainThread | [rest_client:print_UI_logs:2789] Latest logs from UI on 172.23.110.64: 2021-12-02 09:16:44,028 | test | ERROR | MainThread | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638465396725L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:16:36.725Z', u'text': u"Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {'EXIT',<0.8491.142>,\n {{{bad_return_value,{error,timeout}},\n {gen_server,call,\n ['ns_memcached-GleamBookUsers0',\n {delete_vbucket,66},\n 360000]}},\n {gen_server,call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.110.65'},\n {if_rebalance,<0.8373.142>,\n {wait_dcp_data_move,\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.66'],\n 406}},\n infinity]}}}}}.\nRebalance Operation Id = ab0fe02c379c617b89c2ca96499125e2"} 2021-12-02 09:16:44,028 | test | ERROR | MainThread | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638465396709L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:16:36.709Z', u'text': u"Worker <0.8534.142> (for action {move,{406,\n ['ns_1@172.23.110.65',\n 'ns_1@172.23.110.64'],\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.66'],\n []}}) exited with reason {unexpected_exit,\n {'EXIT',\n <0.8491.142>,\n {{{bad_return_value,\n {error,\n timeout}},\n {gen_server,\n call,\n ['ns_memcached-GleamBookUsers0',\n {delete_vbucket,\n 66},\n 360000]}},\n {gen_server,\n call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.110.65'},\n {if_rebalance,\n <0.8373.142>,\n {wait_dcp_data_move,\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.66'],\n 406}},\n infinity]}}}}"} 2021-12-02 09:16:44,028 | test | ERROR | MainThread | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.65', u'tstamp': 1638465396699L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:16:36.699Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.65' disconnected. Check logs for details."} 2021-12-02 09:16:44,029 | test | ERROR | MainThread | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638465114122L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:11:54.122Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-02 09:16:44,029 | test | ERROR | MainThread | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638465113919L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:11:53.919Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-02 09:16:44,029 | test | ERROR | MainThread | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638465113652L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:11:53.652Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.68',\n 'ns_1@172.23.110.66'], EjectNodes = ['ns_1@172.23.110.65'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = ab0fe02c379c617b89c2ca96499125e2"} 2021-12-02 09:16:44,029 | test | ERROR | MainThread | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638465091642L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:11:31.642Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 286 seconds.'} 2021-12-02 09:16:44,029 | test | ERROR | MainThread | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638464759709L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:05:59.709Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1202 08:37:08.710894 20566 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-02 09:16:44,029 | test | ERROR | MainThread | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638464759700L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:05:59.700Z', u'text': u"Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {'EXIT',<0.17831.141>,\n {socket_closed,\n {gen_server,call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.110.64'},\n {if_rebalance,<0.12869.139>,\n {wait_dcp_data_move,\n ['ns_1@172.23.110.68'],\n 66}},\n infinity]}}}}}.\nRebalance Operation Id = 9625a7ee5a3fbbddbac7eeccb99d4b48"} 2021-12-02 09:16:44,029 | test | ERROR | MainThread | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638464759678L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:05:59.678Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-02 09:16:44,032 | infra | INFO | MainThread | [task_manager:shutdown:64] Running TaskManager shutdown ERROR 2021-12-02 09:16:44,684 | test | ERROR | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1640] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'2bd2c283ef5696ca470581e50b195ddf', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=480565efe39ce4c365be1d6aa68de9da', u'status': u'notRunning'} - rebalance failed 2021-12-02 09:16:44,717 | test | INFO | pool-3-thread-26 | [rest_client:print_UI_logs:2789] Latest logs from UI on 172.23.110.68: 2021-12-02 09:16:44,717 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638465396725L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:16:36.725Z', u'text': u"Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {'EXIT',<0.8491.142>,\n {{{bad_return_value,{error,timeout}},\n {gen_server,call,\n ['ns_memcached-GleamBookUsers0',\n {delete_vbucket,66},\n 360000]}},\n {gen_server,call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.110.65'},\n {if_rebalance,<0.8373.142>,\n {wait_dcp_data_move,\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.66'],\n 406}},\n infinity]}}}}}.\nRebalance Operation Id = ab0fe02c379c617b89c2ca96499125e2"} 2021-12-02 09:16:44,717 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638465396709L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:16:36.709Z', u'text': u"Worker <0.8534.142> (for action {move,{406,\n ['ns_1@172.23.110.65',\n 'ns_1@172.23.110.64'],\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.66'],\n []}}) exited with reason {unexpected_exit,\n {'EXIT',\n <0.8491.142>,\n {{{bad_return_value,\n {error,\n timeout}},\n {gen_server,\n call,\n ['ns_memcached-GleamBookUsers0',\n {delete_vbucket,\n 66},\n 360000]}},\n {gen_server,\n call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.110.65'},\n {if_rebalance,\n <0.8373.142>,\n {wait_dcp_data_move,\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.66'],\n 406}},\n infinity]}}}}"} 2021-12-02 09:16:44,717 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.65', u'tstamp': 1638465396699L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:16:36.699Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.65' disconnected. Check logs for details."} 2021-12-02 09:16:44,717 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638465114122L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:11:54.122Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-02 09:16:44,717 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638465113919L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:11:53.919Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-02 09:16:44,717 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638465113652L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:11:53.652Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.68',\n 'ns_1@172.23.110.66'], EjectNodes = ['ns_1@172.23.110.65'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = ab0fe02c379c617b89c2ca96499125e2"} 2021-12-02 09:16:44,717 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638465091642L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:11:31.642Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 286 seconds.'} 2021-12-02 09:16:44,717 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638464759709L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:05:59.709Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1202 08:37:08.710894 20566 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-02 09:16:44,717 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638464759700L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:05:59.700Z', u'text': u"Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {'EXIT',<0.17831.141>,\n {socket_closed,\n {gen_server,call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.110.64'},\n {if_rebalance,<0.12869.139>,\n {wait_dcp_data_move,\n ['ns_1@172.23.110.68'],\n 66}},\n infinity]}}}}}.\nRebalance Operation Id = 9625a7ee5a3fbbddbac7eeccb99d4b48"} 2021-12-02 09:16:44,717 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2791] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1638464759678L, u'shortText': u'message', u'serverTime': u'2021-12-02T09:05:59.678Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-02 09:16:44,719 | test | ERROR | pool-3-thread-26 | [task:call:296] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'2bd2c283ef5696ca470581e50b195ddf', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=480565efe39ce4c365be1d6aa68de9da', u'status': u'notRunning'} - rebalance failed 2021-12-02 09:16:44,719 | infra | INFO | MainThread | [task_manager:shutdown:64] Running TaskManager shutdown 2021-12-02 09:16:44,720 | infra | WARNING | MainThread | [cluster:shutdown:707] Cluster instance shutdown with force 2021-12-02 09:16:44,720 | infra | INFO | MainThread | [task_manager:abort_all_tasks:86] Stopping task Rebalance_task_IN=[]_OUT=[172.23.110.65]_1638465113.59 2021-12-02 09:16:44,720 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:91] Task Rebalance_task_IN=[]_OUT=[172.23.110.65]_1638465113.59 could not be cancelled or already finished. 2021-12-02 09:16:44,721 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-02 09:16:44,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 09:16:45,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-02 09:16:45,086 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 09:16:45,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-02 09:16:45,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-02 09:16:45,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-02 09:16:45,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-02 09:16:45,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-02 09:16:45,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-02 09:16:45,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-02 09:16:45,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 09:16:45,262 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.64: No crash files found 2021-12-02 09:16:45,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 09:16:45,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:45,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:45,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:16:45,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:16:45,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:45,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:45,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:16:45,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:16:45,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:45,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:45,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:16:45,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:16:45,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:45,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:45,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:16:45,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:16:45,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:45,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:45,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:16:45,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:16:45,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:45,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:45,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:16:45,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:16:45,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:45,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:45,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 09:16:45,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 09:16:45,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:45,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:45,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 09:16:45,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-02 09:16:46,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:46,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:46,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 09:16:46,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-02 09:16:46,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:46,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:46,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-02 09:16:46,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-02 09:16:46,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:46,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:46,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-02 09:16:46,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-02 09:16:46,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 09:16:46,244 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-02 09:16:46,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 09:16:46,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:46,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 09:16:46,283 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.64: No 'indexer.log*' files found 2021-12-02 09:16:46,283 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-02 09:16:46,286 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-02 09:16:46,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 09:16:46,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-02 09:16:46,654 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 09:16:46,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-02 09:16:46,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-02 09:16:46,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-02 09:16:46,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-02 09:16:46,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-02 09:16:46,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-02 09:16:46,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-02 09:16:46,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 09:16:46,756 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.65: No crash files found 2021-12-02 09:16:46,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 09:16:46,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:46,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:46,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:16:46,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:16:46,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:46,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:46,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:16:47,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:16:47,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:47,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:47,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:16:47,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:16:47,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:47,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:47,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:16:47,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:16:47,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:47,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:47,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:16:47,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:16:47,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:47,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:47,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:16:47,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:16:47,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:47,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:47,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 09:16:47,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-02 09:16:47,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 09:16:47,444 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-02 09:16:47,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 09:16:47,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:47,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 09:16:47,480 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.65: No 'indexer.log*' files found 2021-12-02 09:16:47,480 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-02 09:16:47,483 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-02 09:16:47,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 09:16:47,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-02 09:16:47,871 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 09:16:47,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-02 09:16:47,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-02 09:16:47,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-02 09:16:47,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-02 09:16:47,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-02 09:16:47,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-02 09:16:47,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-02 09:16:47,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 09:16:47,982 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.66: No crash files found 2021-12-02 09:16:48,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 09:16:48,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:48,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:48,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:16:48,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:16:48,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:48,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:48,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:16:48,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:16:48,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:48,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:48,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:16:48,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:16:48,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:48,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:48,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:16:48,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:16:48,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:48,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:48,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:16:48,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:16:48,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:48,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:48,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:16:48,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-02 09:16:48,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 09:16:48,624 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-02 09:16:48,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 09:16:48,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:48,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 09:16:48,663 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.66: No 'indexer.log*' files found 2021-12-02 09:16:48,663 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-02 09:16:48,665 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-02 09:16:48,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 09:16:49,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-02 09:16:49,178 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 09:16:49,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-02 09:16:49,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-02 09:16:49,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-02 09:16:49,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-02 09:16:49,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-02 09:16:49,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-02 09:16:49,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-02 09:16:49,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 09:16:49,328 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.67: No crash files found 2021-12-02 09:16:49,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 09:16:49,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:49,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:49,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:16:49,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:16:49,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 09:16:49,482 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-02 09:16:49,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 09:16:49,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:49,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 09:16:49,540 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.67: No 'indexer.log*' files found 2021-12-02 09:16:49,542 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-02 09:16:49,545 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-02 09:16:49,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-02 09:16:49,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-02 09:16:49,970 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-02 09:16:49,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-02 09:16:49,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-02 09:16:50,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-02 09:16:50,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-02 09:16:50,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-02 09:16:50,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-02 09:16:50,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-02 09:16:50,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-02 09:16:50,101 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:873] 172.23.110.68: No crash files found 2021-12-02 09:16:50,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-02 09:16:50,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:50,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:50,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:16:50,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-02 09:16:50,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:50,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:50,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:16:50,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-02 09:16:50,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:50,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:50,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:16:50,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-02 09:16:50,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:50,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:50,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:16:50,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-02 09:16:50,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:50,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-02 09:16:50,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:16:50,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-02 09:16:50,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-02 09:16:50,581 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-02 09:16:50,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-02 09:16:50,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-02 09:16:50,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-02 09:16:50,621 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:892] 172.23.110.68: No 'indexer.log*' files found 2021-12-02 09:16:50,621 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-02 09:16:50,621 | test | WARNING | MainThread | [basetestcase:tearDownEverything:583] CLEANUP WAS SKIPPED 2021-12-02 09:16:50,621 | infra | INFO | MainThread | [basetestcase:tearDownEverything:626] ========== tasks in thread pool ========== 2021-12-02 09:16:50,621 | infra | INFO | MainThread | [basetestcase:tearDownEverything:628] ========================================== 2021-12-02 09:16:50,621 | infra | INFO | MainThread | [task_manager:shutdown:64] Running TaskManager shutdown 2021-12-02 09:16:50,621 | infra | INFO | MainThread | [task_manager:shutdown:64] Running TaskManager shutdown 2021-12-02 09:16:50,621 | infra | WARNING | MainThread | [cluster:shutdown:707] Cluster instance shutdown with force ====================================================================== ERROR: SystemTestMagma (aGoodDoctor.Hospital.Murphy) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/aGoodDoctor/Hospital.py", line 495, in SystemTestMagma rebalance_task = self.abort_rebalance(rebalance_task, "kill_memcached") File "pytests/aGoodDoctor/opd.py", line 772, in abort_rebalance reached = RestHelper(rest).rebalance_reached(expected_progress, File "lib/membase/api/rest_client.py", line 97, in rebalance_reached progress = self.rest._rebalance_progress() File "lib/membase/api/rest_client.py", line 1669, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1642, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'2bd2c283ef5696ca470581e50b195ddf', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=480565efe39ce4c365be1d6aa68de9da', u'status': u'notRunning'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 79373.638s FAILEDDuring the test, Remote Connections: 388, Disconnections: 388 SDK Connections: 0, Disconnections: 0 (errors=1) summary so far suite aGoodDoctor.Hospital.Murphy , pass 0 , fail 1 failures so far... aGoodDoctor.Hospital.Murphy.SystemTestMagma testrunner logs, diags and results are available under /data/workspace/magma_temp_job3/logs/testrunner-21-Dec-01_11-13-54/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.9/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 22h 3m 17s 2 actionable tasks: 2 executed [description-setter] Could not determine description. Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Description set: 7.1.0-1817 [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) An attempt to send an e-mail to empty list of recipients, ignored. Finished: UNSTABLE